Distributed%20Compile%20Farm - roybaer/sdcc-wiki GitHub Wiki

Table of Contents Some history Why the old Compile Farm system didn't work The Distributed Compile Farm design The Distributed Compile Farm implementation How to become a Distributed Compile Farm member (builder) How to add a new member (build machine) to Distributed Compile Farm

The SDCC project was painfully stricken by the discontinuation of the SourceForge Compile Farm (SF CF) (see: http://sourceforge.net/forum/forum.php?forum_id=665363). SDCC snapshot builds and regression tests ran on many different platforms on a daily basis, providing to the SDCC development team valuable information about the ability to compile (build) and the correct execution (regression tests) on different platforms. SDCC users had the possibility to take the SDCC snapshot build in which their problem was fixed, without the need to compile the SDCC package by themselves or wait for the next official release, which doesn't happen very often.

We (the SDCC developers) had to find a replacement for the SF CF, so we tried to find volunteers among SDCC developers and SDCC users, who have the possibility and are willing to make snapshot builds on different platforms. We had quite some success, but another problem came on sight: not all candidates were SDCC developers, which means that they didn't have a possibility to access the SourceForge WEB server directly in order to publish the builds on the "SDCC Snapshot Builds" page (see: http://sdcc.sourceforge.net/snap.php).

The basic idea behind the Distributed Compile Farm (DCF) is to have many "builders" having access to a common machine running the DCF Mediator. For an overview of all current "builders" see SDCC Snapshot Builds. The "builder" builds the build on his own machine and sends the build to the DCF server. The DCF Mediator then transfers the build to the SourceForge WEB server:

• every "builder" has his own account and ssh key on the SDCC DCF
• the DCF Mediator periodically scans the builder's home directories and transfers the builds to SourceForge WEB server using the SF ssh key

The DCF Mediator is currently implemented as a shell script (see: http://sdcc.svn.sourceforge.net/viewvc/sdcc/trunk/sdcc-cf/bin/dcf.sh?view=markup), which is invoked from a cron job every 5 minutes. The problem with the current implementation is that all "builder" accounts have to be in the same group as the DCF Mediator. All "builder" files and directories should be searchable and writable by the group members, which is a security hole. The better solution would be to implement the DCF Mediator as a daemon.

NOTE: this chapter is not complete. For further informations send a mail to <sdcc-devel></sdcc-devel>.

• First try to make a snapshot build on the development machine as described at HOWTO Make Snapshot Builds on the Developer Machine.

If you succeed to make the build, than you can proceed with the following steps:

• Install SSH client (probably you already have one if you are on *nix or Linux machine) and create ssh keys for sdcc-builder user as described at SSH Key Generation
• Send the public key and the machine name (which has to be different from any other DFC machine names, listed at SDCC Snapshot Builds) to Erik Petrich
• Erik will reply you when your account will be available
• Try to make the first snapshot build "by hand", to see if everything is OK
• You may create a cron job to make snapshot builds at regular basis. See examples in crontab.txt.

On server running DCF mediator perform the following steps:

• create new account <user></user>, where <user></user> is the new DCF member build machine name
• set the <user></user> primary group to sdcc&amp;amp&#59;amp&amp;&#35;59&#59;&amp;amp&#59;&amp;&#35;35&#59;35&amp;&#35;59&#59;45&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59;builder
• make all files in /home/&amp;amp&#59;amp&amp;&#35;59&#59;lt&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59;user&amp;amp&#59;amp&amp;&#35;59&#59;gt&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59; group writable by executing (as root):

  find /home/&amp&#59;amp&#59&#59;lt&amp&#59;&#35&#59;59&#59&#59;user&amp&#59;amp&#59&#59;gt&amp&#59;&#35&#59;59&#59&#59; &amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;45&amp&#59;&#35&#59;59&#59&#59;type f &amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;45&amp&#59;&#35&#59;59&#59&#59;exec chmod g+rw &amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;123&amp&#59;&#35&#59;59&#59&#59;&amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;125&amp&#59;&#35&#59;59&#59&#59; \&amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;59&amp&#59;&#35&#59;59&#59&#59;

• make all directories in /home/&amp;amp&#59;amp&amp;&#35;59&#59;lt&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59;user&amp;amp&#59;amp&amp;&#35;59&#59;gt&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59; group readable, writable and searchable and set "setgid" permission:

  find /home/&amp&#59;amp&#59&#59;lt&amp&#59;&#35&#59;59&#59&#59;user&amp&#59;amp&#59&#59;gt&amp&#59;&#35&#59;59&#59&#59; &amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;45&amp&#59;&#35&#59;59&#59&#59;type d &amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;45&amp&#59;&#35&#59;59&#59&#59;exec chmod g+rwxs &amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;123&amp&#59;&#35&#59;59&#59&#59;&amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;125&amp&#59;&#35&#59;59&#59&#59; \&amp&#59;amp&#59&#59;&amp&#59;&#35&#59;35&#59&#59;59&amp&#59;&#35&#59;59&#59&#59;

• add the <user></user>'s public ssh key to file /home/&amp;amp&#59;amp&amp;&#35;59&#59;amp&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59;lt&amp;amp&#59;amp&amp;&#35;59&#59;&amp;amp&#59;&amp;&#35;35&#59;35&amp;&#35;59&#59;59&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59;user&amp;amp&#59;amp&amp;&#35;59&#59;amp&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59;gt&amp;amp&#59;amp&amp;&#35;59&#59;&amp;amp&#59;&amp;&#35;35&#59;35&amp;&#35;59&#59;59&amp;amp&#59;&amp;&#35;35&#59;59&amp;&#35;59&#59;/.ssh/authorized_keys

Borut