protocol-1.txt   [plain text]

description of distcc protocol version 1
Copyright (C) 2003 by Martin Pool


This document is provided as explanation for people developing or
debugging distcc.  Discrepancies between this document and the distcc
code are an error in the document.

This document is intended to describe distcc 2.0.

Because this program and document is licensed free of charge, there is
no warranty of any kind, to the extent permitted by applicable law.

If anything is unclear, please ask on the mailing list.


The distcc protocol allows a compiler command line plus C preprocessed
source code to be transmitted from a client computer to a server,
where it is compiled.  The results (object code, exit status, and/or
error messages) are returned to the client.

protocol versioning

This document describes version 1 of the protocol, which is used by
distcc releases up to and including 2.1.  

Future versions of the protocol may vary in any way.

The client begins its request by identifying the protocol version that
it will use.  The server must either respond using the requested
protocol version, or respond with an error in the appropriate protocol
version, or drop the connection.

There is no mechanisms for negotiation of versions.

It is envisaged that when software using a new protocol is released,
users will upgrade both clients and servers at the same time.  Since
distcc is only used within organizations rather than on the internet
this should be realistic.


The distcc protocol runs over a bidirectional non-delimited byte
channel.  This is normally a TCP connection, although operation over
the OpenSSH secure shell is also supported.

Each compilation of a file ("job") corresponds to one invocation of
the client program and connection.

control flow

The overall structure of the protocol is very simple:

  1: The client opens a connection, which is accepted by the server.

  2: The client sends a request.

  3: The server sends a response.

  4: The client and server mutually close the connection.


Connections are initiated by the client.  TCP connections are opened
by connecting to the server port, normally 3632.  SSH connections are
initiated by running the command "distccd --inetd" over SSH.

Either party (client or server) may drop the connection at any point,
in which case the job is discarded by both parties.  In particular,
the server drops the connection if the client's IP address is not
allowed to connect, or if the client makes a protocol error.


The request and response both consist of a series of packets.  Each
packet consists of two or three parts:

  token -- four ascii bytes, defining the type of the packet

  parameter -- eight ascii hexadecimal digits, corresponding to a
  32-bit unsigned quantity 

  body -- optionally present, depending on the token

The sequence in which tokens are sent by both client and server is
always fixed and is specified by this document.

If the type of packet (determined by the token) requires a body, then
the length of the body is given by the parameter.  Otherwise, the
meaning of the parameter depends on the token.

Closing the channel has no meaning and does not indicate the end of a
packet, although it should correspond with the end of the last packet.
An early close indicates an error.

[note: Because the sequence of tokens never varies they are strictly
redundant: each party could just read the parameters and bodies in
order and know what they mean from context.  Sending the tokens
provides a check that both parties understand the data to mean the
same thing and guards against code or network errors.  It also aids
any person examining the network stream.]

[note: Sending the parameter as hexadecimal is redundant, because four
bytes would be sufficient to encode the 32-bit integer.  Sending as
hexadecimal serves as a futher check that the client and server are
synchronized.  Incorrect values for the parameter are detected as a
program or network error.]

[note: The fixed length of the token and length-preceded body allows
IO with a small number of system calls: the header can be read in
exactly twelve bytes, and the body can be read in a single chunk
without needing to parse its contents.]

[note: The length-preceded format means that data cannot be streamed
out until the process that produced it is complete and the length of
the data is known.]


The following packets comprise the request:

DIST <version>
    Greeting from client, followed by the protocol version, which is
    always 1.

ARGC <nargs>

    Specifies the number of arguments in the command, including the
    compiler name.

ARGV <len> <bytes>

    Repeated <nargs> times.  Specifies the string value of one element
    of the command.

DOTI <len> <bytes>

    The contents of the preprocessed source (.i) file.


The following packets comprise the response:

DONE <version>

    Introduces the response.  The version must be 1.

STAT <status>

    Gives the Unix wait status of the compiler.  This 0 for success,
    or otherwise (EXITCODE << 8) | (TERMSIGNAL), depending on whether
    the compiler returned an error or was terminated by a signal.
    (See <sys/wait.h> or a POSIX standard for a full definition.)

SERR <len> <bytes>

    Contains messages sent to stderr by the compiler.

SOUT <len> <bytes>

    Contains messages sent to stdout by the compiler.  Normally empty
    -- errors and warnings for most compilers go to stderr instead.

DOTO <len> <bytes>

    Contains the object file (.o) produced by the compiler.

    In very early versions, this is missing if the status code is not

    In versions up to and including 2.7.1, zero-length output files
    are not supported.  If compilation fails, then the server sends
    the file with a length of zero, but the client does not read it.
    From 2.8, things work as follows:

       regular success:
           status zero
           length nonzero

       succeeded, but zero-byte output:
           status zero          
           length zero
           output file is created/touched

           status nonzero       
           length sent as zero
    This is a little complex but is compatible with earlier releases.

    This does mean we cannot represent a compiler that produces output
    and then fails, but I don't think that is very important.