Valgrind

From PostgreSQL wiki

(Difference between revisions)
Jump to: navigation, search
Line 1: Line 1:
 +
=== Valgrind and Postgres ===
 +
 
[http://http://valgrind.org Valgrind] is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile programs in detail. In particular, Valgrind's Memcheck tool is useful for detecting these bugs. However, it is non-trivial to use with Postgres, and requires modification to Postgres source files to instrument memory allocation and memory context infrastructure with various Valgrind macros.
 
[http://http://valgrind.org Valgrind] is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile programs in detail. In particular, Valgrind's Memcheck tool is useful for detecting these bugs. However, it is non-trivial to use with Postgres, and requires modification to Postgres source files to instrument memory allocation and memory context infrastructure with various Valgrind macros.
  
It is hoped that at some point in the future, Postgres will directly support Valgrind through the use of a configure option, which is possible due to the fact that the header file valgrind.h is under a BSD license, unlike to the rest of Valgrind which is under the GPL 2. In the meantime, this wiki page is the place to obtain an unofficial patch that adds the necessary calls. It is not as comprehensive as it possibly could be, and there are probably other places where specific checks could be usefully injected.   
+
It is hoped that at some point in the future, Postgres will directly support Valgrind through the use of a configure option, which is possible due to the fact that the header file valgrind.h is under a BSD license, unlike the rest of Valgrind which is under the GPL 2. In the meantime, this wiki page is the place to obtain an unofficial patch that adds the necessary calls. It is not as comprehensive as it possibly could be, and there are probably other places where specific checks could be usefully injected.   
  
shared_buffers effectively scrubs memory from Valgrind's perspective.
+
=== The patch itself ===
 +
 
 +
http://wiki.postgresql.org/images/d/d7/Valgrind-hooks-v1.patch.gz
 +
 
 +
Per recommendations in the Valgrind documentation, this patch just copies valgrind.h into the PostgreSQL tree. It is current for the master branch, as of April 3 2012.
  
 
=== General testing procedure ===
 
=== General testing procedure ===
Line 21: Line 27:
 
</source>
 
</source>
  
=== Co-ordination when running tests ===
+
Note that shared_buffers effectively scrubs memory from Valgrind's perspective.
 +
 
 +
== Co-ordination when running tests ==
  
 
postgresql.conf should include a timestamp and PID in log_line_prefix, as well as having a log_min_duration_statement of 0.  Since the valgrind logs included timestamps and were split by PID, they can be used to correlate valgrind errors with particular test suite commands.  Once the test cases yielding valgrind errors are tracked-down, you can rerun the valgrind-ed postmaster with "--track-origins=yes --read-var-info=yes" to get more-specific diagnostics. Valgrind 3.6.0 should be used to get good pinpointing of the error source. At time of writing, version 3.7.0 is the latest stable release.
 
postgresql.conf should include a timestamp and PID in log_line_prefix, as well as having a log_min_duration_statement of 0.  Since the valgrind logs included timestamps and were split by PID, they can be used to correlate valgrind errors with particular test suite commands.  Once the test cases yielding valgrind errors are tracked-down, you can rerun the valgrind-ed postmaster with "--track-origins=yes --read-var-info=yes" to get more-specific diagnostics. Valgrind 3.6.0 should be used to get good pinpointing of the error source. At time of writing, version 3.7.0 is the latest stable release.
  
 
The full installcheck-world run has been found to take something around six hours on a modern machine, but memory consumption is not greatly inflated.  It is recommended that when running Valgrind that you disable CLOBBER_FREED_MEMORY and MEMORY_CONTEXT_CHECKING; they add additional valgrind hook traffic and are redundant with the testing valgrind performs. The patch actually switches the pg_config_manual.h defaults for those settings.
 
The full installcheck-world run has been found to take something around six hours on a modern machine, but memory consumption is not greatly inflated.  It is recommended that when running Valgrind that you disable CLOBBER_FREED_MEMORY and MEMORY_CONTEXT_CHECKING; they add additional valgrind hook traffic and are redundant with the testing valgrind performs. The patch actually switches the pg_config_manual.h defaults for those settings.
 
=== The patch itself ===
 
 
http://wiki.postgresql.org/images/d/d7/Valgrind-hooks-v1.patch.gz
 
 
Per recommendations in the Valgrind documentation, this patch just copies valgrind.h into the PostgreSQL tree. It is current for the master branch, as of April 3 2012.
 

Revision as of 12:25, 4 April 2012

Contents

Valgrind and Postgres

Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile programs in detail. In particular, Valgrind's Memcheck tool is useful for detecting these bugs. However, it is non-trivial to use with Postgres, and requires modification to Postgres source files to instrument memory allocation and memory context infrastructure with various Valgrind macros.

It is hoped that at some point in the future, Postgres will directly support Valgrind through the use of a configure option, which is possible due to the fact that the header file valgrind.h is under a BSD license, unlike the rest of Valgrind which is under the GPL 2. In the meantime, this wiki page is the place to obtain an unofficial patch that adds the necessary calls. It is not as comprehensive as it possibly could be, and there are probably other places where specific checks could be usefully injected.

The patch itself

http://wiki.postgresql.org/images/d/d7/Valgrind-hooks-v1.patch.gz

Per recommendations in the Valgrind documentation, this patch just copies valgrind.h into the PostgreSQL tree. It is current for the master branch, as of April 3 2012.

General testing procedure

For general tests, the recommended procedure is:

# Build Postgres with the valgrind patch
$ cd ~/postgresql
$ patch -p1 < valgrind_postgres.patch
# Building at O1 would probably also be acceptable if O0 proves too slow, but avoid O2
$ ./configure --enable-debug CFLAGS=-"O0 -g"
$ make && make install
# Start Postmaster
$ valgrind --leak-check=no --gen-suppressions=all --suppressions=postgresql/valgrind.supp --time-stamp=yes --log-file=pg-valgrind/%p.log postgres 2>&1 | tee pg-valgrind/postmaster.log
# run tests
$ make installcheck-world

Note that shared_buffers effectively scrubs memory from Valgrind's perspective.

Co-ordination when running tests

postgresql.conf should include a timestamp and PID in log_line_prefix, as well as having a log_min_duration_statement of 0. Since the valgrind logs included timestamps and were split by PID, they can be used to correlate valgrind errors with particular test suite commands. Once the test cases yielding valgrind errors are tracked-down, you can rerun the valgrind-ed postmaster with "--track-origins=yes --read-var-info=yes" to get more-specific diagnostics. Valgrind 3.6.0 should be used to get good pinpointing of the error source. At time of writing, version 3.7.0 is the latest stable release.

The full installcheck-world run has been found to take something around six hours on a modern machine, but memory consumption is not greatly inflated. It is recommended that when running Valgrind that you disable CLOBBER_FREED_MEMORY and MEMORY_CONTEXT_CHECKING; they add additional valgrind hook traffic and are redundant with the testing valgrind performs. The patch actually switches the pg_config_manual.h defaults for those settings.

Personal tools