distcc					-*- indented-text -*-
Martin Pool
$Header: /data/cvs/distcc/src/HACKING,v 1.8 2002/05/16 03:48:53 mbp Exp $


A way to do distributed builds, without needing to do a lot of setup
work in getting NFS source and build directories, identical header
files, and so on.

Designed to be used in conjunction with ccache, which eliminates
redundant compilations, passes through just the preprocessed source to
us and removes preprocessor-specific compiler options.  Perhaps not
strictly required but a very good idea.  (You could perhaps build a .i
file in the Makefile, etc.)

Instead of running the whole compiler on each machine, ship across
the inputs:

  preprocessed source
  command line options

and return the outputs:

  object file
  stdout
  stderr
  return code

Since the point of this is to be fast, we should not do anything
that could potentially slow us down.  Therefore, we have to assume
you're on a secure local network and you can simply open a socket
rather than having to use SSH.

Need to allow parallel builds on remote machines, probably by just
having multiple control channels.  When starting the server, specify
how many children.  It forks that many times and they all try to
accept on the same socket.

You must start the server program on the remote machine.

Question of which remote machine to use when several are available.
Perhaps need to use the 1-byte lock technique to share things out
from the local machine.

It's OK to depend on a modern C library, though ideally it would not
be GNU/Linux specific.  Definitely not x86-specific.

Should sometimes fall through to running the compiler locally.
Probably best to just exec() it directly, passing through argv.

Also needs to work nicely with a local make -j.

This ought to tie in with ccache.  I think invoking ccache first and
having them share preprocessed source would be sufficient.  ccache
passes the precompiled (.i) source to its child.
  
Ideally errors would stream out as compilation continues, but it's
not really required.

On toey, building one C file takes under a second for rsync, but
about 4-6 seconds for Samba.  I wonder what a reasonable transport
time for that much data across SSH would be?

Could have a --check mode that builds the file remotely and also
locally and reports whether they are different.  To be expected if
you're using slightly different compilers or C libraries, but
hopefully not otherwise.

Sun's dmake, for example, requires that the directory be mounted
under the same name from all machines.  On the other hand, it can
run arbitrary makefile commands, not just a C compiler.

  http://sundocs.princeton.edu:8888/ab2/coll.36.5/SWKSHPUSE/@Ab2PageView/idmatch(Z400022C1B141)#Z400022C1B141?=======

I suppose running directly across a socket would be OK if you were
pretty sure the network was safe, which may often be the case.  It's
no worse than NFS.

We need to look for the -c option and run all other commands
locally, because they might be linker invocations and so are not
reliable.

We also need to parse most of the compiler options to work out the
preprocessed and object filenames, because they need to be shipped
across the network.  Can they be recognized just by extension?

If -E (only preprocess) is seen, then we should probably run locally.

We could actually handle -S (do not assemble) remotely by just
adjusting the output filename, but I'm not sure it's common enough to
worry about.

On the remote machine, we need to dump the preprocessed source in a
temporary file, run the compiler, redirecting its output into a
temporary object file, and then send it back.

We might try using sendfile() for shipping bulk data.  For receiving,
we might mmap() the destination file and receive directly into it.

If something fails, then we ought to fall back to running the compiler
locally.  If that fails then just exit nonzero.

Unfortunately we can't really run chrooted because the compiler needs
to access its definition files.  The daemon certainly should refuse to
run as root.

Because gcc is not verified for security wrt it's input, you should
basically consider distccd as giving open slather on the account
running the daemon.

distccd usage:

  -d DIRECTORY		temporary directory, default /tmp/distcc.UID
  -n CHILDREN		number of worker processes
  -p PORT		port to bind

This protocol won't work at all with files over 2GB, but I've never
heard of a source or object file that big.

At the moment if something goes wrong on the server, the client just
gets disconnected and has to compile the file locally.  That's OK, but
it would be nice if the server's error message could also be passed
back.  To do this we probably need a special ecolog handler, and
perhaps for the server to hang around for long enough to pass the
errors back.

We perhaps ought to log the time taken by each compiler invocation.

Interestingly enough, cpp is "10-50% faster" in gcc-3.1 (compared to
3.0), which should help us.

  http://gcc.gnu.org/gcc-3.1/changes.html

=======
When streaming data note that we can't stream *from* a pipe, because
then we won't know how much data is going to come out.  However we can
usefully stream data into a pipe going to the compiler, to overlap
compilation with network transfer.  I guess we could also allow
multiple tokens for each section, which would allow pipes to work.


== COMPRESSION

Preprocessing a simple file takes:
mbp/2 src$ time gcc -E -o serve.i serve.c

real	0m0.026s
user	0m0.030s
sys	0m0.000s
mbp/2 src$ ls -l serve.i serve.c
-rw-r--r--    1 mbp      mbp          5120 May 15 15:57 serve.c
-rw-rw-r--    1 mbp      mbp         78453 May 15 16:00 serve.i

Compressing it takes:

mbp/2 src$ time gzip -3v serve.i
serve.i:	 77.2% -- replaced with serve.i.gz

real	0m0.016s
user	0m0.010s
sys	0m0.000s
-rw-rw-r--    1 mbp      mbp         17917 May 15 16:00 serve.i.gz

Compiling it takes:
mbp/2 src$ time gcc -o serve.o -c serve.i

real	0m0.050s
user	0m0.050s
sys	0m0.000s

So it's marginal: on relatively slow links, the waste of time on the
client is probably made up for by being able to farm out more work.

----
More results using cvs head as of Wed May 15 17:44:47 EST 2002

Building samba head, using plain gcc on anomic:

time make:

with DISTCC_HOSTS='localhost jonquille nevada' make -j4
real	4m4.402s
user	3m41.120s
sys	0m13.460s

just locally:
real	7m15.747s
user	6m40.920s
sys	0m14.460s
