LLVM

Some notes for PostgreSQL developers maintaining LLVM integration.

Cadence

• LLVM branches in January (even major versions) and July (odd major versions), and releases two months later
• LLVM's predictable release calendar began in 2022
• PostgreSQL has a minor release in February, May, August, November
• Fedora is usually the first to upgrade its rolling LLVM version, in April and October, for example LLVM 21, 22, ...
• RHEL soon follows in its next minor release, while Debian and others only adopt new LLVM major versions in their less frequent major releases

This means that we should be testing a new LLVM version and making any required adjustments in January and July. If we don't, and changes turn out to be needed, the Fedora packagers might be blocked, forced to disable JIT, hold back a PostgreSQL minor release, or possible patch locally etc. We might hear about problems sooner from developers using a pre-release Fedora. In some years depending on scheduling coincidences, Debian and other packages might also be affected if we move too slowly.

It's also a good idea to test a bit sooner than that, in case bugs need to be reported that will take some work to fix (example).

In table form, we have:

	LLVM even major version number	LLVM odd major version number
LLVM release branch created	January	July
PostgreSQL minor releases	February	August
LLVM released	March	September
Fedora rolls out new LLVM	April	October
PostgreSQL minor releases	May	November
	June	December

How to test against bleeding-edge LLVM

See instructions for more information, but here is a quick summary. Make sure you have at least a hundred GB of spare disk space. You'll also need lots of RAM and lots of CPU, or failing that, lots of time.

   git clone https://github.com/llvm/llvm-project.git

The default main branch has new development, and release branches are named eg release/22.x.

   src=path/to/llvm-project
   build=path/to/build-dir
   prefix=path/to/install-dir

   cmake \
     -S "$src/llvm" \
     -B "$build" \
     -G Ninja \
     -DCMAKE_C_FLAGS='-gz=zlib' \
     -DCMAKE_CXX_FLAGS='-gz=zlib' \
     -DCMAKE_INSTALL_PREFIX="$prefix" \
     -DCMAKE_BUILD_TYPE="Debug" \
     -DBUILD_SHARED_LIBS="yes" \
     -DLLVM_ENABLE_PROJECTS="clang" \
     -DLLVM_USE_SPLIT_DWARF="yes" \
     -DLLVM_ENABLE_ASSERTIONS="yes" \
     -DLLVM_PARALLEL_LINK_JOBS=16 \

   ninja -C "$build"
   ninja -C "$build" install

• TODO: say which packages you'll need for that to work!
• -DBUILD_SHARED_LIBS="yes" speeds up linking and uses a lot less space, but seems to fail for LLVM < 15 on some systems (perhaps this depends on the system linker being used, not debugged...).
• Ninja will use all the CPUs you have while compiling, but -DLLVM_PARALLEL_LINK_JOBS=16 prevents it from trying to link too many files at once if RAM is limited. 16 seems to be a good number for a system with 128GB, but for smaller systems you might need to turn it down.
• Add -DLLVM_USE_SANITIZER="Address;Undefined" (or variations) to enable sanitizers, but that makes everything even bigger and slower; it may be more useful when you suspect there is a bug in LLVM rather than PostgreSQL, but who knows
• Add -DLLVM_CCACHE_BUILD=ON -DLLVM_CCACHE_DIR="path/to/ccache-dir" if desired, but it doesn't seem to be very effective with large C++ projects due to constantly changing central headers

Now PostgreSQL can be built with:

   LLVM_CONFIG="$prefix/bin/llvm-config" CLANG="$prefix/bin/clang"

If using a sanitizer build, you will also need something like:

   CXXFLAGS="-fsanitize=address -static-libasan" LDFLAGS="-fsanitize=address -static-libasan"

CLANG can also have ccache in front of it. You can also use an external clang of the same major version. In theory bitcode from older clang is supposed to work with newer LLVM too, but we don't recommend it due to lack of testing (and it definitely doesn't work in some cases).

To force maximum use of JIT, the regression tests can be run with these settings in a file whose path is in environment variable TEMP_CONFIG:

   jit_above_cost=0
   jit_optimize_above_cost=0 # 0 = always, -1 = never, good to test both
   jit_inline_above_cost=0 # 0 = always, -1 = never, perhaps ditto?

You might also need to do:

   export LD_LIBRARY_PATH="$prefix/lib"

Packaging/testing do list

• It would be nice to have a build farm animal testing LLVM main branch again (previously that was done by "seawasp")
• It would be nice to have our own package repository for at least Debian that have release, debug+assertion and debug+assertion+sanitizer packages of LLVM, for several architectures, for our full range of supported LLVM versions + bleeding edge nightly build, so that someone can very quickly investigate a problem locally