PostgreSQL Buildfarm Howto

From PostgreSQL wiki

Revision as of 18:57, 29 May 2010 by Mastermind (Talk | contribs)

Jump to: navigation, search

PostgreSQL BuildFarm is a distributed build system designed to detect build failures on a large collection of platforms and configurations. This software is written in Perl. If you're not comfortable with Perl then you possibly don't want to run this, even though the only adjustment you should ever need is to the config file (which is also Perl).

Contents

Get the Software

Download from: pgFoundry Unpack it and put it somewhere. You can put the config file in a different place from the run_build.pl script if you want to, but the simplest thing is to put it in the same place. Decide which user you will run the script as - it must be a user who can run the PostgreSQL server programs (on Unix that means it must *not* run as root). Do everything else here as that user.

Set up a base git mirror that all your branches will pull from.

Most buildfarm members run on more than one branch, and if you do it's good practice to set up a mirror on the buildfarm machine and then just clone that for each branch. As of this writing there are two suitable public git repositories:

The first is the community repository, and is kept very up to date. Unfortunately it is broken on branches earlier then REL8_3_STABLE. The second repository is similar to the first but is not broken on any of the live branches (as of the current writing). When the PostgreSQL project finally moves to using Git, currently expect to be around August 2010, both of these will become redundant. To set up a mirror, do something like this:

 git clone --mirror git://github.com/oicu/pg-cvs-mirror.git pgsql-base.git

When that is done, add an entry to your crontab to keep it up to date, something like:

 20,50 * * * * cd /path/to/pgsql-base.git && git fetch -q

Create a directory where builds will run.

This should be dedicated to the use of the build farm. Make sure there's plenty of space - on my machine each branch can use up to about 700Mb during a build. You can use the directory where the script lives, or a subdirectory of it, or a completely different directory.

Edit the build-farm.conf file

Put the location of the directory you just created in the config variable "build_root". Set the "scm" to "git" and the "scmrepo" to the path to your git mirror. If you are not using the community git repo, set the "scm_url" to point to where to find a givven git commit on the web. e.g. for my mirror on github, the scm_url should be http://github.com/oicu.pg-cvs-mirror/commit/ (Don't forget the trailing "/".) Adjust the config variables "make", "config_opts", and (if you don't use ccache) "config_env" to suit your environment, and to choose which optional postgres coniguration options you want to build with. You should not need to adjust any other variables. Check that you didn't screw things up by running "perl -cw build-farm.conf".

Change the shebang line in the run_build script.

If the path to your perl installation isn't "/usr/bin/perl", edit the #! line in run_build.pl so it is correct. This is the ONLY line in that file you should ever need to edit.

Check that required perl modules are present.

Run "perl -cw run_build.pl". If you get errors about missing perl modules you will need to install them. Most of the required modules are standard modules in any perl distribution.. The rest are all standard CPAN modules, and available either from there or from your OS distribution. When you don't get an error any more, run the same test on run_web_txn.pl. When all is clear you are ready to start testing.

Run in test mode.

With a PATH that matches what you will have when running from cron, run the script in no-send, no-status, verbose mode. Something like this:

 ./run_build.pl --nosend --nostatus --verbose

and watch the fun begin. If this results in failures because it can't find some executables (especially gmake and git), you might need to change the config file again, this time changing the "build_env" with another setting something like:

 PATH => "/usr/local/bin:$ENV{PATH}",

Also, if you put the config file somewhere else, you will need to use the --config=/path/to/build-farm.conf option.

Test running from cron

When you have that running, it's time to try with cron. Put a line in your crontab that looks something like this:

 43 * * * * cd /location/of/run_build.pl/ && ./run_build.pl --nosend --verbose

Again, add the --config option if needed. Notice that this time we didn't specify --nostatus. That means that (after the first run) the script won't do any build work unless the Git repo has changed. Check that your cron job runs (it should email you the results, unless you tell it to send them elsewhere).

Choose which branches you want to build

By default run_build.pl builds the HEAD branch. If you want to build other branches, you can do so by specifying the name on the commandline, e.g.

 run_build.pl REL8_4_STABLE

so, once you have HEAD working, remove the --verbose flag from your crontab, and add extra cron lines for each branch you want to build regularly. You could have something like this:

6 * * * * cd /home/andrew/buildfarm && ./run_build.pl --nosend
30 4 * * * cd /home/andrew/buildfarm && ./run_build.pl --nosend REL8_1_STABLE

Register your new buildfarm member.

Once this is all running happily, you can register to upload your results to the central server. Registration can be done on the buildfarm server at http://www.pgbuildfarm.org/register.html. When you receive your approval by email, you will edit the "animal" and "secret" lines in your config file, remove the --nosend flags, and you are done.

Bugs

Please file bug reports on the tracker at pgFoundry

Running on Windows

There are three build environments for Windows: Cygwin, MinGW/MSys, and Microsoft Visual C++. The buildfarm can run with each of these environments. This section discusses requirements for the buildfarm, rather than requirements for building on Windows, which are covered elsewhere.

Cygwin

There is almost nothing extra to be done for Cygwin. You need to make sure that cygserver is running, and you should set MAX_CONNECTIONS=>3 and CYGWIN=>'server' in the build_env stanza of the buildfarm config. Other than that it should be just like running on Unix.

MinGW/Msys

For MinGW/MSys, you need both the MSys DTK version of perl installed, and a native Windows perl - I have only tested with ActiveState perl, which I have found to be rock solid. You need to run the main buildfarm script using the MSYS DTK perl, and the web transaction script using native Perl. that mean you need to change the first line of the run_web_txn.pl script so it reads something like:

 #!/c/perl/bin/perl

You should make sure that the PATH is set in your config file to put the Native perl ahead of the MSys DTK perl. It's a good idea to have a runbf.bat file that you can call from the Windows scheduler. Mine looks like this:

 @echo off
 setlocal
 c:
 cd \msys\1.0\bin
 c:\msys\1.0\bin\sh.exe --login -c "cd bf && ./run_build.pl --verbose %1 >> bftask.out 2>&1"

Set up a non-privileged Windows user to run this jobs as. set up the buildfarm as above as that user. Then create scheduler jobs that call runbf.bat with an optional branch name argument.

Microsoft Visual C++

For MSVC you need to edit the config file more extensively. Make sure the 'using_msvc' setting is on. Also, there is a section of the file specially for MSVC builds. As with MinGW, you need a native Windows perl installed. It appears that Windows Git does not like to clone local repositories specified with forward slashes (this is pretty horrible - almost all Windows programs are quite happy with forward slashes. Make sure you specify the repository using backslashes or weird things will happen. Again, you will need a runbf.bat file for the windows scheduler. Mine looks like this:

 @echo off
 c:
 cd \prog\bf
 c:\perl\bin\perl run_build.pl --verbose %1 %2 %3 %4 >> bfout.txt

You will also need a tar command capable of bundling up the logs to send to the server. The best one I have found for use on Windows is bsdtar, part of the libarchive collection at http://sourceforge.net/projects/gnuwin32/files/. This is also a good place to get many of the libraries you need for optional pieces of MSVC and MinGW builds.

Personal tools