Profiling with OProfile
From PostgreSQL wiki
OProfile is an operating system level profiler for Linux that's known to give useful results profiling PostgreSQL.
This needs to be done only once per system boot:
sudo opcontrol --init sudo opcontrol --setup --no-vmlinux
although you can re-issue the --setup command to change profiling options later.
If you need detail about what the kernel is doing, get debug symbols for your kernel; then the setup command looks something like
sudo opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux
There are a vast number of other options to the --setup command, but you don't need them for ordinary cases.
Virtual Machine Setup
If you are using a virtual machine, you might need to make oprofile use a timer instead of trying to use hardware performance counters that might not be available. You can do that on the fly by shutting down oprofile if it's running and executing:
sudo opcontrol --deinit sudo modprobe oprofile timer=1
You can make this setting permanent by adding a file to /etc/modprobe.d (at least on Fedora) called oprofile.conf with these contents:
options oprofile timer=1
sudo opcontrol --start sudo opcontrol --reset ... exercise your debug-enabled program here ... sudo opcontrol --dump ; sudo opcontrol --shutdown
The test case should run at least a minute or two to get numbers with reasonable accuracy.
opreport --long-filenames | more opreport -l image:/path/to/postgres | more
If you really want detail:
opannotate --source /path/to/postgres >someplace
The --reset command zeroes the stats that these programs report on, so be sure to save the output you want before running another test case. Also, modifying the executable program-under-test invalidates the stats for it.
Getting callgraph information from oprofile can provide additional detail about libraries that are being used, including using of the common libc library calls. Although it won't directly tell what function in libc is being called, you would see where the calls are coming from, which is usually enough to guess what the libc function is.
Important: to get useful callgraph information, you must compile with -fno-omit-frame-pointer.
You can also get the oprofile data, including callgraph, into kcachegrind, which is *very* helpful, using a script such like http://roberts.vorpus.org/~njs/op2calltree.py
Interpreting the results
A good example showing how to use the results from oprofile to help make a performance-related coding decision is oprofile results for stats collector test
OProfile is preferred to gprof because of several limitations of gprof.
Newer Linux systems (kernels 2.6.31 and later) can use the perf utility.
Installing the software
Whether OProfile works correctly is very dependent on how your Linux distribution was packaged. According to the Unladen-swallow project, "The oProfile packages in Debian Sid and Ubuntu Hardy, Jaunty, Karmic, and Lucid are all broken", with an alternate package suggested there you can swap for the system one. Debian Squeeze should have a working version, as shown in the good Ceph tutorial on using the program. You will need the linux-image-debug package to trace into the kernel, and when that's available you may be able to start the program without the "--no-vmlinux" flag.