PostgreSQL Buildfarm Howto

From PostgreSQL wiki

Jump to: navigation, search

PostgreSQL BuildFarm is a distributed build system designed to detect build failures on a large collection of platforms and configurations. This software is written in Perl. If you're not comfortable with Perl then you possibly don't want to run this, even though the only adjustment you should ever need is to the config file (which is also Perl).


Get the Software

Download from the buildfarm server Unpack it and put it somewhere. You can put the config file in a different place from the script if you want to, but the simplest thing is to put it in the same place. Decide which user you will run the script as - it must be a user who can run the PostgreSQL server programs (on Unix that means it must *not* run as root). Do everything else here as that user.

Other Prerequisites

Must be a new enough version, as some features that are being used were added in version 1.6.
Version 1.7 should work.
Version 1.6 should work.
Version 1.5 will not work.

Choose a setup for a base git mirror that all your branches will pull from.

Most buildfarm members run on more than one branch, and if you do it's good practice to set up a mirror on the buildfarm machine and then just clone that for each branch. The official publicly available git repository is at

and there is a mirror at

Either should be suitable for cloning.

The simplest way to set up a mirror is simply to have the buildfarm script create and maintain it for you. If you do that, the mirror will be updated at the start of a run when it checks to see if any changes have occurred that might require a new build. To do that, all you need to do is set the following two options in your config file:

 git_keep_mirror => 'true',
 git_ignore_mirror_failure => 'true',

If you would rather clone the github mirror for your local mirror instead of the authoritative community repo (doing so can keep load off the community server, which is a good thing), then set the config variable to point to it like this:

 scmrepo => 'git://',

The mirror will be placed at in your build root, above the branch directories.

You can also opt to do create and maintain a git mirror yourself, something like this:

 git clone --mirror git:// pgsql-base.git

When that is done, add an entry to your crontab to keep it up to date, something like:

 20,50 * * * * cd /path/to/pgsql-base.git && git fetch -q

One downside of doing this is that your mirror will only be as up to date as the last time you ran the cron update.

To have your buildfarm installation use a local miror you maintain yourself, set the config variable:

 scmrepo => '/path/to/pgsql-base.git',

Of course, in this case you don't set the git_keep_mirror option.

Create a directory where builds will run.

This should be dedicated to the use of the build farm. Make sure there's plenty of space - on my machine each branch can use up to about 700Mb during a build. You can use the directory where the script lives, or a subdirectory of it, or a completely different directory.

Edit the build-farm.conf file

Notable things you probably need to set include the following:


"git" or "cvs" are the obvious options
Set this to indicate the path to your Git mirror
If you are not using the Community git repository, or want to point the changesets at a different server, set this URL to indicate where to find a given Git commit on the web. For instance, for the github mirror, this value should be: - don't forget the trailing "/".
This will need to be set to the animal you were given by the Buildfarm coordinators
This must be the password indicated by the Buildfarm coordinators

Adjust other config variables "make", "config_opts", and (if you don't use ccache) "config_env" to suit your environment, and to choose which optional postgres configuration options you want to build with.

You should not need to adjust other variables.

You may verify that you didn't screw things up too badly by running "perl -cw build-farm.conf". That verifies that the configuration is still legitimate Perl.

Change the shebang line in the run_build script.

If the path to your perl installation isn't "/usr/bin/perl", edit the #! line in so it is correct. This is the ONLY line in that file you should ever need to edit.

Check that required perl modules are present.

Run "perl -cw". If you get errors about missing perl modules you will need to install them. Most of the required modules are standard modules in any perl distribution. The rest are all standard CPAN modules, and available either from there or from your OS distribution. When you don't get an error any more, run the same test on, and also on if you plan to use that (see below). When all is clear you are ready to start testing.

Run in test mode.

With a PATH that matches what you will have when running from cron, run the script in no-send, no-status, verbose mode. Something like this:

 ./ --nosend --nostatus --verbose

and watch the fun begin. If this results in failures because it can't find some executables (especially gmake and git), you might need to change the config file again, this time changing the "build_env" with another setting something like:

 PATH => "/usr/local/bin:$ENV{PATH}",

Also, if you put the config file somewhere else, you will need to use the --config=/path/to/build-farm.conf option.

If trying to diagnose problems, interesting summary information may be found in the file, which is found in a build-specific directory, of the form $build_root/$CURRENT_BRANCH/$animal.lastrun-logs/

If particular steps of a build failed, logs for those steps may be found in that same directory.

Test running from cron

When you have that running, it's time to try with cron. Put a line in your crontab that looks something like this:

 43 * * * * cd /location/of/ && ./ --nosend --verbose

Again, add the --config option if needed. Notice that this time we didn't specify --nostatus. That means that (after the first run) the script won't do any build work unless the Git repo has changed. Check that your cron job runs (it should email you the results, unless you tell it to send them elsewhere).

Choose which branches you want to build

By default builds the HEAD branch. If you want to build other branches, you can do so by specifying the name on the commandline, e.g. REL8_4_STABLE

so, once you have HEAD working, remove the --verbose flag from your crontab, and add extra cron lines for each branch you want to build regularly. You could have something like this:

6 * * * * cd /home/andrew/buildfarm && ./ --nosend
30 4 * * * cd /home/andrew/buildfarm && ./ --nosend REL8_1_STABLE


There is a wrapper script that makes running multiple branches somewhat easier. To keep build for all the branches that are currently being maintained by the project, uncomment this line in the config file:

 # $conf{branches_to_build} = 'ALL'; # or [qw( HEAD RELx_y_STABLE etc )]

and instead of running, use with the --run-all option. This script accepts all the options that does, and passes them through. So now your crontab could just look like this:

 6 * * * * cd /home/andrew/buildfarm && ./ --run-all

One of the advantages of this is that you don't need to retire a branch when the Postgres project ends support for it, nor to add one when there's a new stable branch. The script contacts the server to get a lits of branches to build that we're interested in, and then builds them. This is now the recommended method of running a buildfarm member.

If you don't want to build all the back branches, you can also use HEAD_PLUS_LATEST or HEAD_PLUS_LATESTn for any n

Register your new buildfarm member.

Once this is all running happily, you can register to upload your results to the central server. Registration can be done on the buildfarm server at When you receive your approval by email, you will edit the "animal" and "secret" lines in your config file, remove the --nosend flags, and you are done.


Please file bug reports on the tracker at pgFoundry

Note also that there is a mailing list about the Buildfarm managed at that project on pgFoundry.

What if you can't use git for some reason?

You can still run in CVS mode against a git-cvs gateway. There is one available for the master (aka HEAD) and REL9_0_STABLE branches.

You will need to be on release 4.2 or later of the buildfarm client, and have the following settings in your config file:

 scm => 'cvs',
 scmrepo => '',
 use_git_cvsserver => 'true',

You will also need to do this, once:

 cvs -d login

An empty password will do.

Using this mode of running is a fall-back. If you can use git you should.

Running on Windows

There are three build environments for Windows: Cygwin, MinGW/MSys, and Microsoft Visual C++. The buildfarm can run with each of these environments. This section discusses requirements for the buildfarm, rather than requirements for building on Windows, which are covered elsewhere.


There is almost nothing extra to be done for Cygwin. You need to make sure that cygserver is running, and you should set MAX_CONNECTIONS=>3 and CYGWIN=>'server' in the build_env stanza of the buildfarm config. Other than that it should be just like running on Unix.


For MinGW/MSys, you need both the MSys DTK version of perl installed, and a native Windows perl - I have only tested with ActiveState perl, which I have found to be rock solid. You need to run the main buildfarm script using the MSYS DTK perl, and the web transaction script using native Perl. that mean you need to change the first line of the script so it reads something like:


You should make sure that the PATH is set in your config file to put the Native perl ahead of the MSys DTK perl. It's a good idea to have a runbf.bat file that you can call from the Windows scheduler. Mine looks like this:

 @echo off
 cd \msys\1.0\bin
 c:\msys\1.0\bin\sh.exe --login -c "cd bf && ./ --verbose %1 >> bftask.out 2>&1"

Set up a non-privileged Windows user to run this jobs as. set up the buildfarm as above as that user. Then create scheduler jobs that call runbf.bat with an optional branch name argument.

Microsoft Visual C++

For MSVC you need to edit the config file more extensively. Make sure the 'using_msvc' setting is on. Also, there is a section of the file specially for MSVC builds. As with MinGW, you need a native Windows perl installed. It appears that Windows Git does not like to clone local repositories specified with forward slashes (this is pretty horrible - almost all Windows programs are quite happy with forward slashes. Make sure you specify the repository using backslashes or weird things will happen. Again, you will need a runbf.bat file for the windows scheduler. Mine looks like this:

 @echo off
 cd \prog\bf
 c:\perl\bin\perl --verbose %1 %2 %3 %4 >> bfout.txt

You will also need a tar command capable of bundling up the logs to send to the server. The best one I have found for use on Windows is bsdtar, part of the libarchive collection at This is also a good place to get many of the libraries you need for optional pieces of MSVC and MinGW builds.

Running multiple buildfarm members on a single machine

Sometimes you might want to run more than one buildfarm member on a single machine. Possible reasons of doing this include testing different compilers, and running with different options. For example, on one FreeBSD machine I have two members, one does a normal build and the other does a build with -DCLOBBER_CACHE_ALWAYS set. Or on a Windows machine one might want to test both the 32 bit and 64 bit mingw-w64 compilers.

The simplest way to do this is to do it all in the same location. Get one member working, then copy the config file to something with the other member's name and change the animal name and password, and whatever in the config will be different from the first one. The members can share a git mirror and build root. There are locking provisions that prevent instances of the buildfarm scripts from tripping over each other. If you are using ccache, you should ensure that each member gets a separate ccache location. The best way to do that is probably to put the member name as well as the branch into the ccache directory name.

Personal tools