https://wiki.postgresql.org/api.php?action=feedcontributions&user=Pgeoghegan&feedformat=atomPostgreSQL wiki - User contributions [en]2024-03-19T02:29:51ZUser contributionsMediaWiki 1.35.13https://wiki.postgresql.org/index.php?title=PostgreSQL_16_Open_Items&diff=37978PostgreSQL 16 Open Items2023-06-10T21:10:05Z<p>Pgeoghegan: Close out "Cleaning up nbtree after logical decoding on standby work"</p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
'''NOTE''': If known, please list the Owner of the open item.<br />
<br />
* Switch to ICU for 17?<br />
** Owner: Jeff Davis<br />
** {{messageLink|82c4c816-06f6-d3e3-ba02-fca4a5cef065@enterprisedb.com|I suggest waiting until next week to commit it and then see what happens}}<br />
** [https://commitfest.postgresql.org/42/4169/ CF Entry]<br />
* {{messageLink|e587e2ee-7de0-88a2-10f8-c7cf001bab8c%40postgrespro.ru|psql: Add role's membership options to the \du+ command}}<br />
** [https://commitfest.postgresql.org/43/4116/ CF Entry]<br />
** NOTE: This is not a committed feature for v16<br />
* {{messageLink|874jp9f5jo.fsf@news-spur.riddles.org.uk|The rules for choosing default ICU locale seem pretty unfriendly}}<br />
** Owner: Jeff Davis<br />
* {{messageLink|ZEZDj1H61ryrmY9o@msg.df7cb.de|could not extend file "base/5/3501" with FileFallocate(): Interrupted system call}}<br />
** Owner: Andres Freund<br />
** Original commit: {{PgCommitURL|4d330a61bb1}}<br />
* {{messageLink|DFBB2D25-DE97-49CA-A60E-07C881EA59A7@winand.at|Inconsistent nulling bitmap in nestloop parameters}}<br />
** Owner: Tom Lane<br />
<br />
== Decisions to Recheck Mid-Beta ==<br />
<br />
* [https://www.postgresql.org/message-id/268fd337-8bb7-92e6-0da2-416c022c11f3%40enterprisedb.com Reconsider a utility_query_id GUC to control if query jumbling of utilities can go through the past string-only mode and the new mode?]<br />
** Potential owner: Michael Paquier<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br />
** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer] [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report] [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report] [https://www.postgresql.org/message-id/flat/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA%40mail.gmail.com Another report] [https://www.postgresql.org/message-id/flat/a53cacb0-8835-57d6-31e4-4c5ef196de1a@deepbluecap.com Another report]<br />
<br />
* [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br />
** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/ZArVOMifjzE7f8W7%40paquier.xyz Requiring recovery.signal or standby.signal when recovering with a backup_label]<br />
** This is a rather old behavior that affects all stable branches, still not something that should be backpatched as-is.<br />
<br />
* {{messageLink|cfcca574-6967-c5ab-7dc3-2c82b6723b99@mail.ru|pg_visibility's pg_check_visible() yields false positive when working in parallel with autovacuum}}<br />
** {{messageLink|1649062270.289865713@f403.i.mail.ru|Thread with patch}} [https://commitfest.postgresql.org/43/3739/ CF Entry]<br />
<br />
* {{messageLink|1516594.1681482708@sss.pgh.pa.us|We are not compatible with newly-released LLVM 16}}<br />
** {{messageLink|CA%2BhUKGKNX_%3Df%2B1C4r06WETKTq0G4Z_7q4L4Fxn5WWpMycDj9Fw%40mail.gmail.com|Patch}}<br />
** Owner: Thomas Munro (volunteer LLVM API change chaser)<br />
<br />
* {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br />
** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|"PANIC: could not open critical system index 2662" - twice}}<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and >MaxAllocSize lengths]<br />
** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br />
** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br />
** Affects all stable branches.<br />
** Fixed at: {{PgCommitURL|8fcb32db98eda1ad2a0c0b40b1cbb5d9a7aa68f0}} and {{PgCommitURL|ffd1b6bb6f8a2ffc929699772610c6925364dbb3}} for HEAD.<br />
<br />
* [https://www.postgresql.org/message-id/flat/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Fix fseek() detection of unseekable files on WIN32]<br />
** Fixed at: {{PgCommitURL|a923e21631a29dc8b8781d7d02b5003d0df64ca3}} and {{PgCommitURL|765f5df726918bcdcfd16bcc5418e48663d1dd59}}, down to 14.<br />
<br />
* {{messageLink|CAAKRu_bETD%2BAri600h6fRjX2p8rJSeMAUp%3D_y88juqOZgouTSg%40mail.gmail.com|Can't disable autovacuum cost delay through storage parameter}}<br />
** Fixed at: {{PgCommitURL|bfac8f8bc4a44c67c9f35b5266676278e4ba1217}}, down to 11.<br />
<br />
* {{messageLink|CAJ7c6TMBTN3rcz4%3DAjYhLPD_w3FFT0Wq_C15jxCDn8U4tZnH1g@mail.gmail.com| EPQ misbehaves for inherited/partitioned tables}}<br />
** Fixed at: {{PgCommitURL|70b42f279}}, down to 14.<br />
<br />
== Non-bugs ==<br />
<br />
* {{messageLink|17862-1ab8f74b0f7b0611@postgresql.org|WindowAgg startup costs don't take into account partition bound. Can lead to incorrect use of cheap startup plans}}<br />
** {{messageLink|CAApHDvrB0S5BMv+0-wTTqWFE-BJ0noWqTnDu9QQfjZ2VSpLv_g@mail.gmail.com|Patch to fix and discussion}}<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 16beta2 ===<br />
* {{messageLink|CAH2-Wz%3D8Z9qY58bjm_7TAHgtW6RzZ5Ke62q5emdCEy9BAzwhmg%40mail.gmail.com|Cleaning up nbtree after logical decoding on standby work}}<br />
** Owner: Peter Geoghegan, Andres Freund<br />
** Original commit: {{PgCommitURL|61b313e4}}<br />
** Fixed at: {{PgCommitURL|d088ba5a}}<br />
* {{messageLink|CAMbWs4_tuVn9EwwMcggGiZJWWstdXX_ci8FeEU17vs+4nLgw3w@mail.gmail.com|Assert failure and wrong query results due to incorrectly removing PHV}}<br />
** Owner: Tom Lane<br />
** Fixed at: {{PgCommitURL|9a2dbc614e6e47da3c49daacec106da32eba9467}}<br />
* {{messageLink|CAMbWs4-_vwkBij4XOQ5ukxUvLgwTm0kS5_DO9CicUeKbEfKjUw%40mail.gmail.com|Assert failure of the cross-check for nullingrels}}<br />
** Owner: Tom Lane<br />
** Original commit: {{PgCommitURL|2489d76c4}}<br />
** [https://commitfest.postgresql.org/43/4250/ CF Entry]<br />
** Fixed at: {{PgCommitURL|991a3df22}}<br />
<br />
=== resolved before 16beta1 ===<br />
* {{messageLink|CAHewXNnu7u1aT%3D%3DWjnCRa%2BSzKb6s80hvwPP_9eMvvvtdyFdqjw%40mail.gmail.com|ERROR: wrong varnullingrels (b 5 7) (expected (b)) for Var 3/3}}<br />
** Fixed at: {{PgCommitURL|d0f952691}}<br />
* {{messageLink|d46f9265-ff3c-6743-2278-6772598233c2%40pgmasters.net|Possible regression setting GUCs on \connect}}<br />
** Owner: Alexander Korotkov<br />
** Discussion on reverting {{PgCommitURL|096dd80f3}}<br />
** Original commit: {{PgCommitURL|096dd80f3}}<br />
** Reverted at: {{PgCommitURL|b9a7a822723aebb16cbe7e5fb874e5124745b07e}}<br />
<br />
* Planner makes improper clause pushdown decisions due to outer-join-aware-Vars changes<br />
** {{messageLink|0b819232-4b50-f245-1c7d-c8c61bf41827@postgrespro.ru|Clause accidentally pushed down}}<br />
** {{messageLink|CAHewXNks3w_Vy9CWoVtHx1XSaeiFpsOzh-zy5eu0Khp1PtG1sA@mail.gmail.com|wrong results due to qual pushdown}}<br />
** Original commit: {{PgCommitURL|2489d76c4}}<br />
** Fixed at: {{PgCommitURL|9df8f903eb6758be5a19e66cdf77e922e9329c31}}<br />
<br />
* Revert {{PgCommitURL|ec386948948}}, per {{messageLink|20230330105325.y6uvpalspynf2frt@alvherre.pgsql|Re: "variable not found in subplan target list"}}<br />
** Reverted at {{PgCommitURL|5472743d9e8}}<br />
<br />
* [https://www.postgresql.org/message-id/CAEZATCWETioXs5kY8vT6BVguY41_wD962VDk%3Du_Nvd7S1UXzuQ%40mail.gmail.com ERROR: ORDER/GROUP BY expression not found in targetlist]<br />
** Fixed at: {{PgCommitURL|da5800d5fa636c6e10c9c98402d872c76aa1c8d0}}<br />
<br />
* [https://www.postgresql.org/message-id/20230212233711.GA1316@telsasoft.com various elogs hit by sqlsmith (ExecRTCheckPerms() and many prunable partitions)]<br />
** Fixed at: {{PgCommitURL|c7468c73f7b6e842a53c12eaee5578a76a8fa7a6}}<br />
<br />
* [https://www.postgresql.org/message-id/20230228235834.GC30529@telsasoft.com pg_dump: zlib compression fails for empty objects (LOs)]<br />
** Fixed at: {{PgCommitURL|00d9dcf5bebbb355152a60f0e2120cdf7f9e7ddd}}<br />
<br />
* [https://www.postgresql.org/message-id/20230227044910.GO1653@telsasoft.com pg_dump: lz4 compression uses no persistent state and writes a block header for every row]<br />
** Fixed at: {{PgCommitURL|0070b66fef21e909adb283f7faa7b1978836ad75}}<br />
<br />
* {{messageLink|3590249.1680971629@sss.pgh.pa.us|Assertion failure with parallel full hash join}}<br />
** Fixed at: {{PgCommitURL|b37d051b0e59e4324e346655a27509507813db79}}<br />
<br />
* {{messageLink|ZDDO6jaESKaBgej0@tamriel.snowman.net|De-revert "Add support for Kerberos credential delegation"}}<br />
** Owner: Stephen Frost<br />
** Original commit: {{PgCommitURL|3d4fa227bce4294ce1cc214b4a9d3b7caa3f0454}}<br />
** Revert: ({{PgCommitURL|3d03b24c350ab060bb223623bdff38835bd7afd0}}<br />
** De-Revert: {{PgCommitURL|6633cfb21691840c33816a6dacaca0b504efb895}}<br />
** Resolved at: {{PgCommitURL|f7431bca8b0138bdbce7025871560d39119565a0}}<br />
<br />
* {{messageLink|c39be3c5-c1a5-1e33-1024-16f527e251a4@enterprisedb.com|SSL tests break on non-existing system CA pool}}<br />
** Fixed at: {{PgCommitURL|0b5d1fb36adda612bd3d5d032463a6eeb0729237}}<br />
<br />
* {{messageLink|CAD21AoBS7o6Ljt_vfqPQPf67AhzKu3fR0iqk8B%3DvVYczMugKMQ%40mail.gmail.com|VacuumUpdateCosts() logging condition incorrect for some initial values of vacuum_cost_delay}}<br />
** Fixed at: {{PgCommitURL|a9781ae11ba2fdb44a3a72c9a7ebb727140b25c5}}<br />
<br />
* {{messageLink|CA%2BhUKGJ-ZPJwKHVLbqye92-ZXeLoCHu5wJL6L6HhNP7FkJ%3DmeA%40mail.gmail.com|check_strxfrm_bug()}}<br />
** Owner: Thomas Munro<br />
** Fixed at: {{PgCommitURL|7d3d72b55edd1b7552a9a358991555994efab0e9}}<br />
<br />
* {{messageLink|20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de|Should we remove vacuum_defer_cleanup_age?}}<br />
** Owner: Andres Freund<br />
** Fixed at: {{PgCommitURL|1118cd37eb61e6a2428f457a8b2026a7bb3f801a}}<br />
<br />
* {{messageLink|2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com|Add two missing tests in 035_standby_logical_decoding.pl}}<br />
** Fixed at: {{PgCommitURL|376dc820531bafcbf105fff74c5b14c23d9950af}}<br />
** Fixed at: {{PgCommitURL|a6e04b1d20c2e9cece9b64bb5b36ebfdc3a9031b}}<br />
<br />
* {{messageLink|b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com|Performance regression due to SQLValueFunction removal}}<br />
** Fixed at: {{PgCommitURL|d8c3106bb60e4f87be595f241e173ba3c2b7aa2c}}<br />
<br />
* {{messageLink|20230419172326.dhgyo4wrrhulovt6%40awork3.anarazel.de|pg_stat_io not tracking smgrwriteback() is confusing}}<br />
** Owner: Andres Freund<br />
** Fixed at: {{PgCommitURL|093e5c57d506783a95dd8feddd9a3f2651e1aeba}}<br />
<br />
* {{messageLink|ZFhCyn4Gm2eu60rB@paquier.xyz|Table data compression is broken with pg_dump --compress lz4}}<br />
** Owner: Tomas Vondra<br />
** Fixed at: {{PgCommitURL|1a05c1d252993b0a59c58a6daf91a2df9333044f}}<br />
<br />
* {{messageLink|94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com|Valgrind unhappy with LZ4F code in pg_dump}}<br />
** Owner: Tomas Vondra<br />
** Fixed at: {{PgCommitURL|3c18d90f8907e53c3021fca13ad046133c480e4d}}<br />
<br />
* {{messageLink|20230509190247.3rrplhdgem6su6cg@awork3.anarazel.de|walsender performance regression due to logical decoding on standby changes}}<br />
** Owner: Andres Freund<br />
** Original commit: {{PgCommitURL|e101dfac}}<br />
** Fixed at: {{PgCommitURL|bc971f4025c378ce500d86597c34b0ef996d4d8c}}<br />
<br />
== Won't Fix ==<br />
<br />
* Is it OK that WL_SOCKET_ACCEPT is less fair on Windows than on Unix (and than the coding before 16) when there are multiple server sockets configured?<br />
** {{messageLink|CA%2BhUKG%2BA2dk29hr5zRP3HVJQ-_PncNJM6HVQ7aaYLXLRBZU-xw%40mail.gmail.com|WL_SOCKET_ACCEPT fairness on Windows}} has a (blind) patch to fix that, but would need a Windows hacker to test<br />
** Owner: Thomas Munro<br />
** Original commit: {{PgCommitURL|7389aad6}}<br />
** Issue reclassified as a non-critical improvement to be [https://commitfest.postgresql.org/43/4263/ considered for 17]<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Beta 2: TBD<br />
* Beta 1: May 25, 2023<br />
* Feature Freeze: April 8, 2023 0:00 AoE ('''Last Day to Commit Features''')<br />
<br />
== See also ==<br />
<br />
* [[Release Management Team]]<br />
* [[PostgreSQL 15 Open Items]]<br />
<br />
[[Category:Open_Items]]</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Committing_checklist&diff=37863Committing checklist2023-05-24T20:23:34Z<p>Pgeoghegan: Add advice about commit.template file</p>
<hr />
<div>This document is an attempt to list common checks that PostgreSQL project [[Committers]] may want to adopt as part of a checklist of things to check before pushing. There are certain classic mistakes that even experienced committers have been known to make occasionally. In the real world, many mistakes happen when a step is skipped over during a routine process, perhaps caused by a seemingly insignificant last minute change. It's important to learn from these mistakes.<br />
<br />
This checklist isn't intended as something that committers will adopt wholesale. Rather, it is intended as a starting point for creating your own semi-customized checklist. Since your final checklist is supposed to be used more or less mechanically, it shouldn't ever be too long, and should be organized into sections to make it easier to skip items where irrelevant. In short, if it's worth adopting something as a standard practice that you return to again and again, it's probably also worth writing that down, to formalize it. Use discretion when deciding what makes sense for you.<br />
<br />
= Basic checks =<br />
<br />
* Double-check release build compiler warnings.<br />
<br />
* make check-world.<br />
** You may want to speed this up by using the following recipe:<br />
make -j16 -s install;make -Otarget -j10 -s check-world && echo "quick make-check world success" || echo "quick make-check world failure"<br />
<br />
* Consider the need for a catversion bump.<br />
<br />
* Don't assume that you haven't broken the doc build if you make even a trivial doc change.<br />
** Removing a GUC can break instances in the release notes where they're referenced. <br />
** Even grep can miss this, since references to the GUC will have dashes rather than underscores, plus possibly other variations.<br />
<br />
* Validate err*() calls against https://www.postgresql.org/docs/devel/static/error-style-guide.html<br />
<br />
* Validate *printf calls for trailing newlines.<br />
<br />
* Spellcheck the patch.<br />
<br />
* Verify that long lines are not better broken into several shorter lines:<br />
git diff origin/master | grep -E '^(\+|diff)' | sed 's/^+//' | expand -t4 | awk "length > 78 || /^diff/"<br />
<br />
* Run pgindent, pgperltidy, and reformat-dat-files on changed files; keep the changes minimal.<br />
<br />
* Run pgperlcritic on modified Perl files.<br />
<br />
* Update version numbers, if needed:<br />
CATALOG_VERSION_NO, PG_CONTROL_VERSION, XLOG_PAGE_MAGIC, PGSTAT_FILE_FORMAT_ID<br />
<br />
* Update function/other OIDs, if needed;<br />
<br />
= Regression test checks =<br />
<br />
* When adding core regression test files, make sure that they're added to both serial and parallel schedules.<br />
(But release 14 and later have only the parallel schedule.)<br />
<br />
* Look for alternative output files for any regression test you're updating the output of.<br />
** Some tests have alternative output files to work around portability issues.<br />
** Most of the time it works to just apply the same patch to the other variants as the delta you're observing for the output file that's relevant to your own platform.<br />
** Occasionally you may have to just see what the buildfarm says.<br />
<br />
= Git checks =<br />
<br />
== Basic ==<br />
<br />
* Do a dry run before really pushing by using --dry-run.<br />
<br />
* Look at "git status"; anything missing?<br />
<br />
* Author and committer timestamps should match.<br />
<br />
This can be an issue if you're in the habit of rebasing, or apply a patch with "git am". Make sure that your setup displays both in "git log", by specifying "--pretty=fuller", or changing the git format config. The easiest way to make both timestamps match is to amend the commit like so:<br />
<br />
git commit --amend --reset-author<br />
<br />
If you have "autosetuprebase = always" in your git config, then a last minute "git pull" could cause a rebase, which could cause author and committer timestamps to diverge a bit. In practice, small differences between author and committer timestamp are not considered to be a problem.<br />
<br />
* Write log message (consider creating a [https://www.git-scm.com/docs/git-commit/2.38.0#Documentation/git-commit.txt--tltfilegt .gitmessage commit.template template file] to make this easier):<br />
Discussion: https://postgr.es/m/XXXXXXXXXXX<br />
Back-patch depth?<br />
What should the release notes say?<br />
Credit any reviewer.<br />
<br />
* When making references to other commits, it's a good idea to use the first 9 chars of the commit SHA. Fewer than 9 means there will be no hyperlink in the HTTP interface. More than 9 is not required.<br />
<br />
* Note compatibility issues in commit message, so that they'll get picked up later, when release notes are written.<br />
<br />
* Check merge with master (not applicable to commits).<br />
<br />
* If you're using a dedicated ssh key with a passphrase, you may find it useful to deliberately disable it when you're done pushing:<br />
<br />
$ ssh-add -d ~/.ssh/id_rsa_postgres<br />
<br />
== Backpatching and git ==<br />
<br />
Commit messages for multiple branches should be identical when back-patching, in order to have tooling recognize the redundancy for purposes of compiling release notes, and other things of that nature.<br />
<br />
* Easiest way to get commit metadata consistent is to not worry about commit messages outside of the master branch at first. Commit message on backbranches could initially be something like "pending 9.6".<br />
<br />
* Perform the following procedure on each back branch when you're done, by checking out each individual branch in gitmaster local clone, and doing this for master branch commit which has good commit message:<br />
<br />
git commit --amend --reset-author -C <commit><br />
<br />
You now have the same commit message on each branch. This means that the <code>src/tools/git_changelog</code> utility script will present the commits from each affected local branch together, as one logical change. (This script is used as a starting point when writing back branch release notes. Note that the concept of "one logical change" is not a standard git concept.)<br />
<br />
* Use <code>git push origin : --dry-run</code> to dry-run pushing all branches at once. Once satisfied, remove --dry-run to actually push. --dry-run is doubly important if you push each branch individually.<br />
<br />
= Maintaining ABI compatibility while backpatching =<br />
<br />
Avoid breaking ABI compatibility. It's unacceptable for extensions built against an earlier point release to break in a more recent point release.<br />
<br />
* You can only really change the signature of a function with local linkage, perhaps with a few rare exceptions.<br />
* You cannot modify any struct definition in src/include/*. If any new members must be added to a struct, put them at the end in backbranches. It's okay to have a different struct layout in master. Even then, extensions that allocate the struct can break via a dependency on its size.<br />
* Move new enum values to the end.<br />
<br />
See [https://postgr.es/m/1315116.1603900649@sss.pgh.pa.us this message] for more considerations on ABI preservation.<br />
<br />
= GUC checks =<br />
<br />
* When adding a new GUC, postgresql.conf.sample needs to be updated, too.<br />
<br />
* Is the GUC group the right one?<br />
<br />
= Advanced smoke tests =<br />
<br />
* Valgrind memcheck + "make installcheck".<br />
<br />
* CLOBBER_CACHE_ALWAYS.<br />
<br />
* When doing anything that touches WAL-logging, consider creating a replica, and making sure that wal_consistency_checking=all passes on replica while master runs "make installcheck". WAL_DEBUG makes any bug that this throws up easier to isolate.<br />
<br />
* "#define COPY_PARSE_PLAN_TREES" and "#define WRITE_READ_PARSE_PLAN_TREES" can catch omissions or other mistakes when "src/backend/nodes/*" were changed.<br />
<br />
* Various tests that are only run on certain platforms, enabled [https://www.postgresql.org/docs/devel/regress-run.html using PG_TEST_EXTRA or EXTRA_TESTS environment variables]. For example, PG_TEST_EXTRA='ssl' and EXTRA_TESTS='collate.linux.utf8' tests.<br />
<br />
* check for unaligned access with things from c.h like -fsanitize=alignment<br />
<br />
* sqlsmith (for grammar changes, and ??)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Committing_checklist&diff=37862Committing checklist2023-05-24T20:15:21Z<p>Pgeoghegan: Remove obsolete guidance on testing older branches</p>
<hr />
<div>This document is an attempt to list common checks that PostgreSQL project [[Committers]] may want to adopt as part of a checklist of things to check before pushing. There are certain classic mistakes that even experienced committers have been known to make occasionally. In the real world, many mistakes happen when a step is skipped over during a routine process, perhaps caused by a seemingly insignificant last minute change. It's important to learn from these mistakes.<br />
<br />
This checklist isn't intended as something that committers will adopt wholesale. Rather, it is intended as a starting point for creating your own semi-customized checklist. Since your final checklist is supposed to be used more or less mechanically, it shouldn't ever be too long, and should be organized into sections to make it easier to skip items where irrelevant. In short, if it's worth adopting something as a standard practice that you return to again and again, it's probably also worth writing that down, to formalize it. Use discretion when deciding what makes sense for you.<br />
<br />
= Basic checks =<br />
<br />
* Double-check release build compiler warnings.<br />
<br />
* make check-world.<br />
** You may want to speed this up by using the following recipe:<br />
make -j16 -s install;make -Otarget -j10 -s check-world && echo "quick make-check world success" || echo "quick make-check world failure"<br />
<br />
* Consider the need for a catversion bump.<br />
<br />
* Don't assume that you haven't broken the doc build if you make even a trivial doc change.<br />
** Removing a GUC can break instances in the release notes where they're referenced. <br />
** Even grep can miss this, since references to the GUC will have dashes rather than underscores, plus possibly other variations.<br />
<br />
* Validate err*() calls against https://www.postgresql.org/docs/devel/static/error-style-guide.html<br />
<br />
* Validate *printf calls for trailing newlines.<br />
<br />
* Spellcheck the patch.<br />
<br />
* Verify that long lines are not better broken into several shorter lines:<br />
git diff origin/master | grep -E '^(\+|diff)' | sed 's/^+//' | expand -t4 | awk "length > 78 || /^diff/"<br />
<br />
* Run pgindent, pgperltidy, and reformat-dat-files on changed files; keep the changes minimal.<br />
<br />
* Run pgperlcritic on modified Perl files.<br />
<br />
* Update version numbers, if needed:<br />
CATALOG_VERSION_NO, PG_CONTROL_VERSION, XLOG_PAGE_MAGIC, PGSTAT_FILE_FORMAT_ID<br />
<br />
* Update function/other OIDs, if needed;<br />
<br />
= Regression test checks =<br />
<br />
* When adding core regression test files, make sure that they're added to both serial and parallel schedules.<br />
(But release 14 and later have only the parallel schedule.)<br />
<br />
* Look for alternative output files for any regression test you're updating the output of.<br />
** Some tests have alternative output files to work around portability issues.<br />
** Most of the time it works to just apply the same patch to the other variants as the delta you're observing for the output file that's relevant to your own platform.<br />
** Occasionally you may have to just see what the buildfarm says.<br />
<br />
= Git checks =<br />
<br />
== Basic ==<br />
<br />
* Do a dry run before really pushing by using --dry-run.<br />
<br />
* Look at "git status"; anything missing?<br />
<br />
* Author and committer timestamps should match.<br />
<br />
This can be an issue if you're in the habit of rebasing, or apply a patch with "git am". Make sure that your setup displays both in "git log", by specifying "--pretty=fuller", or changing the git format config. The easiest way to make both timestamps match is to amend the commit like so:<br />
<br />
git commit --amend --reset-author<br />
<br />
If you have "autosetuprebase = always" in your git config, then a last minute "git pull" could cause a rebase, which could cause author and committer timestamps to diverge a bit. In practice, small differences between author and committer timestamp are not considered to be a problem.<br />
<br />
* Write log message:<br />
Discussion: https://postgr.es/m/XXXXXXXXXXX<br />
Back-patch depth?<br />
What should the release notes say?<br />
Credit any reviewer.<br />
<br />
* When making references to other commits, it's a good idea to use the first 9 chars of the commit SHA. Fewer than 9 means there will be no hyperlink in the HTTP interface. More than 9 is not required.<br />
<br />
* Note compatibility issues in commit message, so that they'll get picked up later, when release notes are written.<br />
<br />
* Check merge with master (not applicable to commits).<br />
<br />
* If you're using a dedicated ssh key with a passphrase, you may find it useful to deliberately disable it when you're done pushing:<br />
<br />
$ ssh-add -d ~/.ssh/id_rsa_postgres<br />
<br />
== Backpatching and git ==<br />
<br />
Commit messages for multiple branches should be identical when back-patching, in order to have tooling recognize the redundancy for purposes of compiling release notes, and other things of that nature.<br />
<br />
* Easiest way to get commit metadata consistent is to not worry about commit messages outside of the master branch at first. Commit message on backbranches could initially be something like "pending 9.6".<br />
<br />
* Perform the following procedure on each back branch when you're done, by checking out each individual branch in gitmaster local clone, and doing this for master branch commit which has good commit message:<br />
<br />
git commit --amend --reset-author -C <commit><br />
<br />
You now have the same commit message on each branch. This means that the <code>src/tools/git_changelog</code> utility script will present the commits from each affected local branch together, as one logical change. (This script is used as a starting point when writing back branch release notes. Note that the concept of "one logical change" is not a standard git concept.)<br />
<br />
* Use <code>git push origin : --dry-run</code> to dry-run pushing all branches at once. Once satisfied, remove --dry-run to actually push. --dry-run is doubly important if you push each branch individually.<br />
<br />
= Maintaining ABI compatibility while backpatching =<br />
<br />
Avoid breaking ABI compatibility. It's unacceptable for extensions built against an earlier point release to break in a more recent point release.<br />
<br />
* You can only really change the signature of a function with local linkage, perhaps with a few rare exceptions.<br />
* You cannot modify any struct definition in src/include/*. If any new members must be added to a struct, put them at the end in backbranches. It's okay to have a different struct layout in master. Even then, extensions that allocate the struct can break via a dependency on its size.<br />
* Move new enum values to the end.<br />
<br />
See [https://postgr.es/m/1315116.1603900649@sss.pgh.pa.us this message] for more considerations on ABI preservation.<br />
<br />
= GUC checks =<br />
<br />
* When adding a new GUC, postgresql.conf.sample needs to be updated, too.<br />
<br />
* Is the GUC group the right one?<br />
<br />
= Advanced smoke tests =<br />
<br />
* Valgrind memcheck + "make installcheck".<br />
<br />
* CLOBBER_CACHE_ALWAYS.<br />
<br />
* When doing anything that touches WAL-logging, consider creating a replica, and making sure that wal_consistency_checking=all passes on replica while master runs "make installcheck". WAL_DEBUG makes any bug that this throws up easier to isolate.<br />
<br />
* "#define COPY_PARSE_PLAN_TREES" and "#define WRITE_READ_PARSE_PLAN_TREES" can catch omissions or other mistakes when "src/backend/nodes/*" were changed.<br />
<br />
* Various tests that are only run on certain platforms, enabled [https://www.postgresql.org/docs/devel/regress-run.html using PG_TEST_EXTRA or EXTRA_TESTS environment variables]. For example, PG_TEST_EXTRA='ssl' and EXTRA_TESTS='collate.linux.utf8' tests.<br />
<br />
* check for unaligned access with things from c.h like -fsanitize=alignment<br />
<br />
* sqlsmith (for grammar changes, and ??)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37769Meson2023-04-19T16:56:27Z<p>Pgeoghegan: /* Test related commands */</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| <code>./configure [<i>options</i>]</code><br />
|| <code>meson setup [<i>options</i>] [<i>builddir</i>] <i>sourcedir</i></code><br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| <code>perl src/tools/msvc/mkvcbuild.pl</code><br />
|| <code>meson setup --backend vs [<i>options</i>] [<i>builddir</i>] <i>sourcedir</i></code><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| <code>./configure --help</code><br />
|| <code>meson configure</code><br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| <code>./configure --prefix=<i>DIR</i>, --$somedir=<i>DIR</i>, --with-$option, --enable-$feature</code><br />
|| <code>meson setup|configure -D$option=$value</code><br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| <code>--enable-cassert</code><br />
|| <code>-Dcassert=true</code><br />
||<br />
|-<br />
|| enable debug symbols<br />
|| <code>./configure --enable-debug</code><br />
|| <code>meson configure|setup -Ddebug=true</code><br />
||<br />
|-<br />
|| specify compiler<br />
|| <code>CC=<i>compiler</i> ./configure</code><br />
|| <code>CC=<i>compiler</i> meson setup</code><br />
|| <code>CC</code> is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| <code>CFLAGS=<i>options</i> ./configure</code><br />
|| <code>meson configure|setup -Dc_args=<i>options</i></code><br />
|| <code>CFLAGS</code> is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| <code>make -s</code><br />
|| <code>ninja</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| <code>make</code><br />
|| <code>ninja -v</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| <code>make install</code><br />
|| <code>ninja install</code><br />
|| use <code>meson install --quiet</code> for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| <code>meson install --only-changed</code><br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| <code>make clean</code><br />
|| <code>ninja clean</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| <code>cd doc/ && make html && make man</code><br />
|| <code>ninja docs</code><br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the <code>-C</code> flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| <code>meson test --list</code><br />
|| Only shows tests from "tmp_install" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup], since it is the default (<code>--setup tmp_install</code> is implied here)<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| <code>meson test --setup running --list</code><br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| <code>make check-world</code><br />
|| <code>meson test -v</code><br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| <code>make installcheck-world</code><br />
|| <code>meson test -v --setup running</code><br />
|| Currently [https://postgr.es/m/CAH2-Wz=X7=5jU-+XXJaqQRZja_fseEtrd_dGJa0Wpb74OpsgEA@mail.gmail.com makes brittle assumptions] about test libraries being installed<br />
|-<br />
|| run main regression tests<br />
|| <code>make check</code><br />
|| <code>meson test -v --suite setup --suite regress</code><br />
|| <code>--suite setup</code> required to get a <code>tmp_install</code> directory; see below<br />
|-<br />
|| run specific contrib test suite<br />
|| <code>make -C contrib/amcheck check</code><br />
|| <code>meson test -v --suite setup --suite amcheck</code><br />
|| <code>--suite setup</code> required to get a <code>tmp_install</code> directory; see below<br />
|-<br />
|| run main regression tests against existing server<br />
|| <code>make installcheck</code><br />
|| <code>meson test -v --setup running --suite regress-running</code><br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| <code>make -C contrib/amcheck installcheck</code><br />
|| <code>meson test -v --setup running --suite amcheck-running</code><br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
When running a specific test suite against a temporary throw away installation, <code>--suite setup</code> should generally be specified. Otherwise the tests could end up running against a stale <code>tmp_install</code> directory, causing general confusion. This [https://postgr.es/m/20230209205605.zo5gfhli22g2kdm2@awork3.anarazel.de workaround] is not required when running tests against an existing server (via the <code>running</code> test setup and variant test suites), since of course the installation directory being tested is whatever directory the external server installation uses.<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be omitted] when using a reasonably recent meson version (meson 0.46 or later), which we assume here.<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
As documented on the [https://mesonbuild.com/Getting-meson.html meson website], a MSI installer is also available.<br />
<br />
Using the most recent version of ActivePerl may be a bit challenging, as there is no direct access to a "perl" command except if enabling a project registered in the ActivePerl website, with a command like that:<br />
<pre><br />
state activate --default<br />
</pre><br />
<br />
An easy way to set up things is to install Chocolatey, and rely on StrawberryPerl. Here are the main packages to worry about:<br />
<pre><br />
choco install winflexbison<br />
choco install sed<br />
choco install gzip<br />
choco install strawberryperl<br />
choco install diffutils<br />
</pre><br />
<br />
The compiler detected will depend on the Command Prompt type used. For MSVC, use the command prompt installed for VS. A native Command prompt or Powershell may finish by linking to Chocolatey's gcc, which may be OK, still be careful with what's reported by meson setup.<br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37768Meson2023-04-19T16:48:57Z<p>Pgeoghegan: Comments about dependency issue belong under plain "--suite regress-running", not under "meson test -v --setup running --suite regress-running"</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| <code>./configure [<i>options</i>]</code><br />
|| <code>meson setup [<i>options</i>] [<i>builddir</i>] <i>sourcedir</i></code><br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| <code>perl src/tools/msvc/mkvcbuild.pl</code><br />
|| <code>meson setup --backend vs [<i>options</i>] [<i>builddir</i>] <i>sourcedir</i></code><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| <code>./configure --help</code><br />
|| <code>meson configure</code><br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| <code>./configure --prefix=<i>DIR</i>, --$somedir=<i>DIR</i>, --with-$option, --enable-$feature</code><br />
|| <code>meson setup|configure -D$option=$value</code><br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| <code>--enable-cassert</code><br />
|| <code>-Dcassert=true</code><br />
||<br />
|-<br />
|| enable debug symbols<br />
|| <code>./configure --enable-debug</code><br />
|| <code>meson configure|setup -Ddebug=true</code><br />
||<br />
|-<br />
|| specify compiler<br />
|| <code>CC=<i>compiler</i> ./configure</code><br />
|| <code>CC=<i>compiler</i> meson setup</code><br />
|| <code>CC</code> is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| <code>CFLAGS=<i>options</i> ./configure</code><br />
|| <code>meson configure|setup -Dc_args=<i>options</i></code><br />
|| <code>CFLAGS</code> is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| <code>make -s</code><br />
|| <code>ninja</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| <code>make</code><br />
|| <code>ninja -v</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| <code>make install</code><br />
|| <code>ninja install</code><br />
|| use <code>meson install --quiet</code> for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| <code>meson install --only-changed</code><br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| <code>make clean</code><br />
|| <code>ninja clean</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| <code>cd doc/ && make html && make man</code><br />
|| <code>ninja docs</code><br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the <code>-C</code> flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| <code>meson test --list</code><br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| <code>meson test --setup running --list</code><br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| <code>make check-world</code><br />
|| <code>meson test -v</code><br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| <code>make installcheck-world</code><br />
|| <code>meson test -v --setup running</code><br />
|| Currently [https://postgr.es/m/CAH2-Wz=X7=5jU-+XXJaqQRZja_fseEtrd_dGJa0Wpb74OpsgEA@mail.gmail.com makes brittle assumptions] about test libraries being installed<br />
|-<br />
|| run main regression tests<br />
|| <code>make check</code><br />
|| <code>meson test -v --suite setup --suite regress</code><br />
|| <code>--suite setup</code> required to get a <code>tmp_install</code> directory; see below<br />
|-<br />
|| run specific contrib test suite<br />
|| <code>make -C contrib/amcheck check</code><br />
|| <code>meson test -v --suite setup --suite amcheck</code><br />
|| <code>--suite setup</code> required to get a <code>tmp_install</code> directory; see below<br />
|-<br />
|| run main regression tests against existing server<br />
|| <code>make installcheck</code><br />
|| <code>meson test -v --setup running --suite regress-running</code><br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| <code>make -C contrib/amcheck installcheck</code><br />
|| <code>meson test -v --setup running --suite amcheck-running</code><br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
When running a specific test suite against a temporary throw away installation, <code>--suite setup</code> should generally be specified. Otherwise the tests could end up running against a stale <code>tmp_install</code> directory, causing general confusion. This [https://postgr.es/m/20230209205605.zo5gfhli22g2kdm2@awork3.anarazel.de workaround] is not required when running tests against an existing server (via the <code>running</code> test setup and variant test suites), since of course the installation directory being tested is whatever directory the external server installation uses.<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be omitted] when using a reasonably recent meson version (meson 0.46 or later), which we assume here.<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
As documented on the [https://mesonbuild.com/Getting-meson.html meson website], a MSI installer is also available.<br />
<br />
Using the most recent version of ActivePerl may be a bit challenging, as there is no direct access to a "perl" command except if enabling a project registered in the ActivePerl website, with a command like that:<br />
<pre><br />
state activate --default<br />
</pre><br />
<br />
An easy way to set up things is to install Chocolatey, and rely on StrawberryPerl. Here are the main packages to worry about:<br />
<pre><br />
choco install winflexbison<br />
choco install sed<br />
choco install gzip<br />
choco install strawberryperl<br />
choco install diffutils<br />
</pre><br />
<br />
The compiler detected will depend on the Command Prompt type used. For MSVC, use the command prompt installed for VS. A native Command prompt or Powershell may finish by linking to Chocolatey's gcc, which may be OK, still be careful with what's reported by meson setup.<br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37767Meson2023-04-19T16:41:58Z<p>Pgeoghegan: Document assumption that test libraries are available from install directory with "--setup running", which has been broken since commit b6a0d469ca</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| <code>./configure [<i>options</i>]</code><br />
|| <code>meson setup [<i>options</i>] [<i>builddir</i>] <i>sourcedir</i></code><br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| <code>perl src/tools/msvc/mkvcbuild.pl</code><br />
|| <code>meson setup --backend vs [<i>options</i>] [<i>builddir</i>] <i>sourcedir</i></code><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| <code>./configure --help</code><br />
|| <code>meson configure</code><br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| <code>./configure --prefix=<i>DIR</i>, --$somedir=<i>DIR</i>, --with-$option, --enable-$feature</code><br />
|| <code>meson setup|configure -D$option=$value</code><br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| <code>--enable-cassert</code><br />
|| <code>-Dcassert=true</code><br />
||<br />
|-<br />
|| enable debug symbols<br />
|| <code>./configure --enable-debug</code><br />
|| <code>meson configure|setup -Ddebug=true</code><br />
||<br />
|-<br />
|| specify compiler<br />
|| <code>CC=<i>compiler</i> ./configure</code><br />
|| <code>CC=<i>compiler</i> meson setup</code><br />
|| <code>CC</code> is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| <code>CFLAGS=<i>options</i> ./configure</code><br />
|| <code>meson configure|setup -Dc_args=<i>options</i></code><br />
|| <code>CFLAGS</code> is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| <code>make -s</code><br />
|| <code>ninja</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| <code>make</code><br />
|| <code>ninja -v</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| <code>make install</code><br />
|| <code>ninja install</code><br />
|| use <code>meson install --quiet</code> for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| <code>meson install --only-changed</code><br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| <code>make clean</code><br />
|| <code>ninja clean</code><br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| <code>cd doc/ && make html && make man</code><br />
|| <code>ninja docs</code><br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the <code>-C</code> flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| <code>meson test --list</code><br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| <code>meson test --setup running --list</code><br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| <code>make check-world</code><br />
|| <code>meson test -v</code><br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| <code>make installcheck-world</code><br />
|| <code>meson test -v --setup running</code><br />
||<br />
|-<br />
|| run main regression tests<br />
|| <code>make check</code><br />
|| <code>meson test -v --suite setup --suite regress</code><br />
|| <code>--suite setup</code> required to get a <code>tmp_install</code> directory; see below<br />
|-<br />
|| run specific contrib test suite<br />
|| <code>make -C contrib/amcheck check</code><br />
|| <code>meson test -v --suite setup --suite amcheck</code><br />
|| <code>--suite setup</code> required to get a <code>tmp_install</code> directory; see below<br />
|-<br />
|| run main regression tests against existing server<br />
|| <code>make installcheck</code><br />
|| <code>meson test -v --setup running --suite regress-running</code><br />
|| Currently [https://postgr.es/m/CAH2-Wz=X7=5jU-+XXJaqQRZja_fseEtrd_dGJa0Wpb74OpsgEA@mail.gmail.com makes brittle assumptions] about test libraries being installed<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| <code>make -C contrib/amcheck installcheck</code><br />
|| <code>meson test -v --setup running --suite amcheck-running</code><br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
When running a specific test suite against a temporary throw away installation, <code>--suite setup</code> should generally be specified. Otherwise the tests could end up running against a stale <code>tmp_install</code> directory, causing general confusion. This [https://postgr.es/m/20230209205605.zo5gfhli22g2kdm2@awork3.anarazel.de workaround] is not required when running tests against an existing server (via the <code>running</code> test setup and variant test suites), since of course the installation directory being tested is whatever directory the external server installation uses.<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be omitted] when using a reasonably recent meson version (meson 0.46 or later), which we assume here.<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
As documented on the [https://mesonbuild.com/Getting-meson.html meson website], a MSI installer is also available.<br />
<br />
Using the most recent version of ActivePerl may be a bit challenging, as there is no direct access to a "perl" command except if enabling a project registered in the ActivePerl website, with a command like that:<br />
<pre><br />
state activate --default<br />
</pre><br />
<br />
An easy way to set up things is to install Chocolatey, and rely on StrawberryPerl. Here are the main packages to worry about:<br />
<pre><br />
choco install winflexbison<br />
choco install sed<br />
choco install gzip<br />
choco install strawberryperl<br />
choco install diffutils<br />
</pre><br />
<br />
The compiler detected will depend on the Command Prompt type used. For MSVC, use the command prompt installed for VS. A native Command prompt or Powershell may finish by linking to Chocolatey's gcc, which may be OK, still be careful with what's reported by meson setup.<br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=PgCon_2023_Developer_Meeting&diff=37760PgCon 2023 Developer Meeting2023-04-15T23:38:15Z<p>Pgeoghegan: /* RSVPs */</p>
<hr />
<div>A meeting of the interested PostgreSQL developers is being planned for Tuesday 30 May, 2023 at the University of Ottawa, prior to pgCon 2023. In order to keep the numbers manageable, this meeting is by '''invitation only'''.<br />
Any questions regarding the invitations to this event should be directed to the team of individuals tasked with coming up with the list of people to invite:<br />
<br />
* Andres Freund<br />
* Stephen Frost<br />
* Dave Page<br />
<br />
An Unconference will be held on Friday for in-depth discussion of technical topics.<br />
<br />
This is a PostgreSQL Community event.<br />
<br />
== Meeting Goals ==<br />
<br />
* Define the schedule for the upcoming releases<br />
* Address any proposed timing, policy, or procedure issues<br />
* Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.<br />
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br />
<br />
== Time & Location ==<br />
<br />
The meeting will (probably) be:<br />
<br />
* 9:00AM to 12PM<br />
* DMS 3105 - Desmarais Hall, 55 Laurier Avenue East<br />
* University of Ottawa.<br />
<br />
Lunch will be served during the meeting.<br />
<br />
== COVID-19 ==<br />
<br />
The University of Ottawa's COVID-19 guidance can be found at https://www.uottawa.ca/en/covid-19. Wearing of masks at the Developer Meeting will be optional, however we do ask that people do not attend if they have COVID symptoms or have tested positive.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!<br />
<br />
# Joe Conway<br />
# Jeff Davis<br />
# Peter Eisentraut<br />
# Andres Freund<br />
# Stephen Frost<br />
# Etsuro Fujita<br />
# Peter Geoghegan<br />
# Magnus Hagander<br />
# Jonathan Katz<br />
# Alexander Korotkov<br />
# Tom Lane<br />
# Heikki Linnakangas<br />
# Noah Misch<br />
# Thomas Munro<br />
# Dave Page<br />
# Michael Paquier<br />
# Melanie Plageman<br />
# David Rowley<br />
# Tomas Vondra<br />
<br />
The following people will not be in Ottawa, and do not plan to attend:<br />
<br />
# Masao Fujii<br />
# Daniel Gustafsson<br />
# Tatsuo Ishii<br />
# Dean Rasheed<br />
<br />
== Agenda Items ==<br />
<br />
* 16.0 release and commitfest schedule (Dave)<br />
* Improvements to table AM API (Alexander)<br />
* ''Please add suggestions for agenda items here. (with your name)''<br />
<br />
==Agenda==<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Item<br />
!Presenter<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|09:00 - 09:10<br />
|Welcome and introductions<br />
|Dave Page<br />
<br />
|- <br />
|09:10 - 09:20<br />
|Release and commitfest schedules<br />
|Dave Page<br />
<br />
|- <br />
|??:?? - ??:??<br />
|TBD<br />
|TBD<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|10:30 - 11:00<br />
|Coffee break<br />
|All<br />
<br />
|- <br />
|??:?? - ??:??<br />
|TBD<br />
|TBD<br />
<br />
|- <br />
|11:50 - 12:00<br />
|Any other business<br />
|Dave Page<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|12:00<br />
|Lunch<br />
|<br />
<br />
|}<br />
<br />
Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move materially however). Any remaining time before lunch may be used for Commitfest item triage or other activities.<br />
<br />
[[Category:Developer Meeting]]</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=PgCon_2023_Developer_Meeting&diff=37727PgCon 2023 Developer Meeting2023-04-10T16:26:57Z<p>Pgeoghegan: /* RSVPs */</p>
<hr />
<div>A meeting of the interested PostgreSQL developers is being planned for Tuesday 30 May, 2023 at the University of Ottawa, prior to pgCon 2023. In order to keep the numbers manageable, this meeting is by '''invitation only'''.<br />
Any questions regarding the invitations to this event should be directed to the team of individuals tasked with coming up with the list of people to invite:<br />
<br />
* Andres Freund<br />
* Stephen Frost<br />
* Dave Page<br />
<br />
An Unconference will be held on Friday for in-depth discussion of technical topics.<br />
<br />
This is a PostgreSQL Community event.<br />
<br />
== Meeting Goals ==<br />
<br />
* Define the schedule for the upcoming releases<br />
* Address any proposed timing, policy, or procedure issues<br />
* Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.<br />
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br />
<br />
== Time & Location ==<br />
<br />
The meeting will (probably) be:<br />
<br />
* 9:00AM to 12PM<br />
* DMS 3105 - Desmarais Hall, 55 Laurier Avenue East<br />
* University of Ottawa.<br />
<br />
Lunch will be served during the meeting.<br />
<br />
== COVID-19 ==<br />
<br />
The University of Ottawa's COVID-19 guidance can be found at https://www.uottawa.ca/en/covid-19. Wearing of masks at the Developer Meeting will be optional, however we do ask that people do not attend if they have COVID symptoms or have tested positive.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!<br />
<br />
# Joe Conway<br />
# Jeff Davis<br />
# Peter Eisentraut<br />
# Andres Freund<br />
# Stephen Frost<br />
# Peter Geoghegan<br />
# Etsuro Fujita<br />
# Magnus Hagander<br />
# Jonathan Katz<br />
# Alexander Korotkov<br />
# Tom Lane<br />
# Heikki Linnakangas<br />
# Noah Misch<br />
# Thomas Munro<br />
# Dave Page<br />
# Michael Paquier<br />
# Melanie Plageman<br />
# David Rowley<br />
# Tomas Vondra<br />
<br />
The following people will not be in Ottawa, and do not plan to attend:<br />
<br />
# Masao Fujii<br />
# Daniel Gustafsson<br />
# Tatsuo Ishii<br />
# Dean Rasheed<br />
<br />
== Agenda Items ==<br />
<br />
* 16.0 release and commitfest schedule (Dave)<br />
* ''Please add suggestions for agenda items here. (with your name)''<br />
<br />
==Agenda==<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Item<br />
!Presenter<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|09:00 - 09:10<br />
|Welcome and introductions<br />
|Dave Page<br />
<br />
|- <br />
|09:10 - 09:20<br />
|Release and commitfest schedules<br />
|Dave Page<br />
<br />
|- <br />
|??:?? - ??:??<br />
|TBD<br />
|TBD<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|10:30 - 11:00<br />
|Coffee break<br />
|All<br />
<br />
|- <br />
|??:?? - ??:??<br />
|TBD<br />
|TBD<br />
<br />
|- <br />
|11:50 - 12:00<br />
|Any other business<br />
|Dave Page<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|12:00<br />
|Lunch<br />
|<br />
<br />
|}<br />
<br />
Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move materially however). Any remaining time before lunch may be used for Commitfest item triage or other activities.<br />
<br />
[[Category:Developer Meeting]]</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=PgCon_2023_Developer_Meeting&diff=37708PgCon 2023 Developer Meeting2023-04-04T18:39:25Z<p>Pgeoghegan: Add my name</p>
<hr />
<div>A meeting of the interested PostgreSQL developers is being planned for Tuesday 30 May, 2023 at the University of Ottawa, prior to pgCon 2023. In order to keep the numbers manageable, this meeting is by '''invitation only'''.<br />
Any questions regarding the invitations to this event should be directed to the team of individuals tasked with coming up with the list of people to invite:<br />
<br />
* Andres Freund<br />
* Stephen Frost<br />
* Dave Page<br />
<br />
An Unconference will be held on Friday for in-depth discussion of technical topics.<br />
<br />
This is a PostgreSQL Community event.<br />
<br />
== Meeting Goals ==<br />
<br />
* Define the schedule for the upcoming releases<br />
* Address any proposed timing, policy, or procedure issues<br />
* Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.<br />
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br />
<br />
== Time & Location ==<br />
<br />
The meeting will (probably) be:<br />
<br />
* 9:00AM to 12PM<br />
* DMS 3105 - Desmarais Hall, 55 Laurier Avenue East<br />
* University of Ottawa.<br />
<br />
Lunch will be served during the meeting.<br />
<br />
== COVID-19 ==<br />
<br />
The University of Ottawa's COVID-19 guidance can be found at https://www.uottawa.ca/en/covid-19. Wearing of masks at the Developer Meeting will be optional, however we do ask that people do not attend if they have COVID symptoms or have tested positive.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!<br />
<br />
# Joe Conway<br />
# Jeff Davis<br />
# Peter Eisentraut<br />
# Andres Freund<br />
# Stephen Frost<br />
# Etsuro Fujita<br />
# Magnus Hagander<br />
# Jonathan Katz<br />
# Alexander Korotkov<br />
# Tom Lane<br />
# Heikki Linnakangas<br />
# Noah Misch<br />
# Thomas Munro<br />
# Dave Page<br />
# Michael Paquier<br />
# Melanie Plageman<br />
# David Rowley<br />
# Peter Geoghegan<br />
<br />
The following people will not be in Ottawa, and do not plan to attend:<br />
<br />
# Masao Fujii<br />
# Daniel Gustafsson<br />
# Tatsuo Ishii<br />
# Dean Rasheed<br />
<br />
== Agenda Items ==<br />
<br />
* 16.0 release and commitfest schedule (Dave)<br />
* ''Please add suggestions for agenda items here. (with your name)''<br />
<br />
==Agenda==<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Item<br />
!Presenter<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|09:00 - 09:10<br />
|Welcome and introductions<br />
|Dave Page<br />
<br />
|- <br />
|09:10 - 09:20<br />
|Release and commitfest schedules<br />
|Dave Page<br />
<br />
|- <br />
|??:?? - ??:??<br />
|TBD<br />
|TBD<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|10:30 - 11:00<br />
|Coffee break<br />
|All<br />
<br />
|- <br />
|??:?? - ??:??<br />
|TBD<br />
|TBD<br />
<br />
|- <br />
|11:50 - 12:00<br />
|Any other business<br />
|Dave Page<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|12:00<br />
|Lunch<br />
|<br />
<br />
|}<br />
<br />
Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move materially however). Any remaining time before lunch may be used for Commitfest item triage or other activities.<br />
<br />
[[Category:Developer Meeting]]</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD&diff=37582Getting a stack trace of a running PostgreSQL backend on Linux/BSD2023-02-19T00:44:09Z<p>Pgeoghegan: /* Jumping back and forth through a recording use GDB commands */</p>
<hr />
<div>[[Generating a stack trace of a PostgreSQL backend|Up to parent]]<br />
<br />
== Linux and BSD ==<br />
<br />
Linux and BSD systems generally use the [http://gcc.gnu.org/ GNU compiler collection] and the [http://www.gnu.org/software/gdb/ GNU Debugger] ("gdb"). It's pretty trivial to get a stack trace of a process.<br />
<br />
(If you want more than just a stack trace, take a look at the [[Developer FAQ]] which covers interactive debugging).<br />
<br />
=== Installing External symbols ===<br />
<br />
(BSD users who installed from ports can skip this)<br />
<br />
On many Linux systems, debugging info is separated out from program binaries and stored separately. It's often not installed when you install a package, so if you want to debug the program (say, get a stack trace) you will need to install debug info packages. Unfortunately, the names of these packages vary depending on your distro, as does the procedure for installing them.<br />
<br />
Some generic instructions (unrelated to PostgreSQL) are maintained on the GNOME Wiki [http://live.gnome.org/GettingTraces/DistroSpecificInstructions here].<br />
<br />
==== On Debian ====<br />
<br />
http://wiki.debian.org/HowToGetABacktrace<br />
<br />
Debian Squeeze (6.x) users will also need to install gdb 7.3 from backports, as the gdb shipped in Squeeze doesn't understand the PIE executables used in newer PostgreSQL builds.<br />
<br />
==== On Ubuntu ====<br />
<br />
First, follow the instructions on the Ubuntu wiki entry [https://wiki.edubuntu.org/DebuggingProgramCrash DebuggingProgramCrash]. <br />
<br />
Once you've finished enabling the use of debug info packages as described, you will need to use the <code>list-dbgsym-packages.sh</code> script linked to on that wiki article to get a list of debug packages you need. Installing the debug package for postgresql alone is <i>not</i> sufficient. <br />
<br />
After following the instructions on the Ubuntu wiki, download the script to your desktop, open a terminal, and run:<br/><br />
<pre><br />
$ sudo apt-get install $(sudo bash Desktop/list-dbgsym-packages.sh -t -p $(pidof -s postgres))<br />
</pre><br />
<br />
==== On Fedora ====<br />
<br />
All Fedora versions: [https://fedoraproject.org/wiki/StackTraces#debuginfo FedoraProject.org - StackTraces]<br />
<br />
==== Other distros ====<br />
<br />
In general, you need to install at least the debug symbol packages for the PostgreSQL server and client as well as any common package that may exist, and the debug symbol package for libc. It's a good idea to add debug symbols for the other libraries PostgreSQL uses in case the problem you're having arises in or touches on one of those libraries.<br />
<br />
=== Collecting a stack trace ===<br />
<br />
==== How to tell if a stack trace is any good ====<br />
<br />
Read this section and keep it in mind as you collect information using the instructions below. Making sure the information you collect is actually useful will save you, and everybody else, time and hassle.<br />
<br />
It is vitally important to have debugging symbols available to get a useful stack trace. If you do not have the required symbols installed, backtraces will contain lots of entries like this:<br />
<br />
<pre><br />
#1 0x00686a3d in ?? ()<br />
#2 0x00d3d406 in ?? ()<br />
#3 0x00bf0ba4 in ?? ()<br />
#4 0x00d3663b in ?? ()<br />
#5 0x00d39782 in ?? ()<br />
</pre><br />
<br />
... which are completely useless for debugging without access to your system (and almost useless with access). If you see results like the above, you need to install debugging symbol packages, or even re-build postgresql with debugging enabled. <b>Do not bother collecting such backtraces, they are not useful.</b><br />
<br />
Sometimes you'll get backtraces that contain just the function name and the executable it's within, not source code file names and line numbers or parameters. Such output will have lines like this:<br />
<br />
<pre><br />
#11 0x00d3afbe in PostmasterMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
</pre><br />
<br />
This isn't ideal, but is a lot better than nothing. Installing debug information packages should give an even more detailed stack trace with line number and argument information, like this:<br />
<br />
<pre><br />
#9 0xb758d97e in PostmasterMain (argc=5, argv=0xb813a0e8) at postmaster.c:1040<br />
</pre><br />
<br />
... which is the most useful for tracking down your problem. Note the reference to a source file and line number instead of just an executable name.<br />
<br />
==== Identifying the backend to connect to ====<br />
<br />
You need to know the process ID of the postgresql backend to connect to. If you're interested in a backend that's using lots of CPU it might show up in <code>top</code>. If you have a current connection to the backend you're interested in, use <code>select pg_backend_pid()</code> to get its process ID. Otherwise, the <code>pg_catalog.pg_stat_activity</code> and/or <code>pg_catalog.pg_locks</code> views may be useful in identifying the backend of interest; see the "procpid" column in those views.<br />
<br />
==== Attaching gdb to the backend ====<br />
<br />
Once you know the process ID to connect to, run:<br />
<br />
<pre><br />
sudo gdb -p pid<br />
</pre><br />
<br />
where "pid" is the process ID of the backend. GDB will pause the execution of the process you specified and drop you into interactive mode (the <code>(gdb)</code> prompt) after showing the call the backend is currently running, eg:<br />
<br />
<pre><br />
0xb7c73424 in __kernel_vsyscall ()<br />
(gdb) <br />
</pre><br />
<br />
You'll want to tell gdb to save a log of the session to a file, so at the gdb prompt enter:<br />
<br />
<pre><br />
(gdb) set pagination off<br />
(gdb) set logging file debuglog.txt<br />
(gdb) set logging on<br />
</pre><br />
<br />
gdb is now saving all input and output to a file, <code>debuglog.txt</code>, in the directory in which you started gdb.<br />
<br />
At this point execution of the backend is still paused. It can even hold up other backends, so I recommend that you tell it to resume executing normally with the "cont" command:<br />
<br />
<pre><br />
(gdb) cont<br />
Continuing.<br />
</pre><br />
<br />
The backend is now running normally, as if gdb wasn't connected to it.<br />
<br />
==== Getting the trace ====<br />
<br />
OK, with gdb connected you're ready to get a useful stack trace.<br />
<br />
In addition to the instructions below, you can find some useful tips about using gdb with postgresql backends on the [[Developer_FAQ#What_debugging_features_are_available.3F|Developer FAQ]].<br />
<br />
==== Getting representative traces from a running backend ====<br />
<br />
If you're concerned with a case that's taking way too long to execute a query, is using too much CPU, or appears to be in an infinite loop, you'll want to <i>repeatedly</i> interrupt its execution, get a stack trace, and let it resume executing. Having a collection of several stack traces helps provide a better idea of where it's spending its time.<br />
<br />
You interrupt the backend and get back to the gdb command line with ^C (control-C). Once at the gdb command line, you use the "bt" command to get a backtrace, then the "cont" command to resume normal backend execution.<br />
<br />
Once you've collected a few backtraces, detach then exit gdb at the gdb interactive prompt:<br />
<br />
<pre><br />
(gdb) detach<br />
Detaching from program: /usr/lib/postgresql/8.3/bin/postgres, process 12912<br />
(gdb) quit<br />
user@host:~$<br />
</pre><br />
<br />
An alternative approach is to use the <code>gcore</code> program to save a series of core dumps of the running program without disrupting its execution. Those core dumps may then be examined at your leisure, giving you time to get more than just a backtrace because you're not holding up the backend's execution while you think and type.<br />
<br />
==== Getting a trace from the point of an error report ====<br />
<br />
If you are trying to find out the cause of an unexpected error, the most useful thing to do is to set a breakpoint at '''errfinish''' before you let the backend continue:<br />
<br />
<pre><br />
(gdb) b errfinish<br />
Breakpoint 1 at 0x80ced0: file elog.c, line 414.<br />
(gdb) cont<br />
Continuing.<br />
</pre><br />
<br />
Now, in your connected psql session, run whatever query is needed to provoke the error. When it happens, the backend will stop execution at '''errfinish'''.<br />
Collect your backtrace with '''bt''', then '''quit''' (or, possibly, '''cont''' if you want to do it again).<br />
<br />
A breakpoint at '''errfinish''' will capture generation of not only ERROR reports, but also NOTICE, LOG, and any other message that isn't suppressed by '''client_min_messages'''<br />
or '''log_min_messages'''. You may want to adjust those settings to avoid having to continue through a bunch of unrelated messages.<br />
<br />
==== Getting a trace from a reproducibly crashing backend ====<br />
<br />
GDB will automatically interrupt the execution of a program if it detects a crash. So, once you've attached gdb to the backend you expect to crash, you just let it continue execution as normal and do whatever you need to to make the backend crash.<br />
<br />
gdb will drop you into interactive mode as the backend crashes. At the <code>gdb</code> prompt you can enter the <code>bt</code> command to get a stack trace of the crash, then <code>cont</code> to continue execution. When gdb reports the process has exited, use the <code>quit</code> command.<br />
<br />
Alternately, you can collect a core file as explained below, but it's probably more hassle than it's worth if you know which backend to attach gdb to before it crashes.<br />
<br />
==== Getting a trace from a randomly crashing backend ====<br />
<br />
It's a lot harder to get a stack trace from a backend that's crashing when you don't know why it's crashing, what causes a backend to crash, or which backends will crash when. For this, you generally need to enable the generation of core files, which are debuggable dumps of a program's state that are generated by the operating system when the program crashes. <br />
<br />
===== Enabling core dumps =====<br />
<br />
[http://www.cyberciti.biz/tips/linux-core-dumps.html This article provides a useful primer on core dumps on Linux].<br />
<br />
On a Linux system you can check to see if core file generation is enabled for a process by examining /proc/$pid/limits, where $pid is the process ID of interest. "Max core file size" should be non-zero.<br />
<br />
Generally, adding "ulimit -c unlimited" to the top of the PostgreSQL startup script and restarting postgresql is sufficient to enable core dump collection. Make sure you have plenty of free space in your PostgreSQL data directory, because that's where the core dumps will be written and they can be fairly big due to Pg's use of shared memory. It may be useful to <b>temporarily reduce the size of shared_buffers</b> within postgresql.conf. This avoids core dumps that make the system unresponsive for minutes at a time, which can happen when shared_buffers is more than a few gigabytes. Reducing shared_buffers significantly will usually not make the server intolerably slow, since PostgreSQL will make increased use of the filesystem cache.<br />
<br />
On a Linux system it's also worth changing the file name format used for core dumps so that core dumps don't overwrite each other. The <code>/proc/sys/kernel/core_pattern</code> file controls this. I suggest <code>core.%p.sig%s.%ts</code>, which will record the process's PID, the signal that killed it, and the timestamp at which the core was generated. See <code>man 5 core</code>. To apply the settings change just run <code>echo core.%p.sig%s.%ts | sudo tee -a /proc/sys/kernel/core_pattern</code>.<br />
<br />
You can test whether core dumps are enabled by starting a `psql' session, finding the backend pid for it using the instructions given above, then killing it with "kill -ABRT pidofbackend" (where pidofbackend is the PID of the postgres backend, NOT the pid of psql). You should see a core file appear in your postgresql data directory.<br />
<br />
===== Debugging the core dump =====<br />
<br />
Once you've enabled core dumps, you need to wait until you see a backend crash. A core dump will be generated by the operating system, and you'll be able to attach gdb to it to collect a stack trace or other information. <br />
<br />
You need to tell gdb what executable file generated the core if you want to get useful backtraces and other debugging information. To do this, just specify the postgres executable path then the core file path when invoking gdb, as shown below. If you do not know the location of the postgres executable, you can get it by examining /proc/$pid/exe for a running postgres instance. For example:<br />
<br />
<pre><br />
$ for f in `pgrep postgres`; do ls -l /proc/$f/exe; done<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:30 /proc/10621/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11052/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11053/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11054/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11055/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
</pre><br />
<br />
... we can see from the above that the postgres executable on my (Ubuntu) system is <code>/usr/lib/postgresql/8.4/bin/postgres</code>.<br />
<br />
Once you know the executable path and the core file location, just run gdb with those as arguments, ie <code>gdb -q /path/to/postgres /path/to/core</code>. Now you can debug it as if it was a normal running postgres, as discussed in the sections above.<br />
<br />
===== Debugging the core dump - example =====<br />
<br />
For example, having just forced a postgres backend to crash with <code>kill -ABRT</code>, I have a core file named <code>core.10780.sig6.1271644870s</code> in <code>/var/lib/postgresql/8.4/main</code>, which is the data directory on my Ubuntu system. I've used /proc to find out that the executable for postgres on my system is <code>/usr/lib/postgresql/8.4/bin/postgres</code>.<br />
<br />
It's now easy to run GDB against it and request a backtrace:<br />
<br />
<pre><br />
$ sudo -u postgres gdb -q -c /var/lib/postgresql/8.4/main/core.10780.sig6.1271644870s /usr/lib/postgresql/8.4/bin/postgres<br />
Core was generated by `postgres: wal writer process '.<br />
Program terminated with signal 6, Aborted.<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
(gdb) bt<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
#1 0x00686a3d in ___newselect_nocancel () from /lib/tls/i686/cmov/libc.so.6<br />
#2 0x00e68d25 in pg_usleep () from /usr/lib/postgresql/8.4/bin/postgres<br />
#3 0x00d3d406 in WalWriterMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
#4 0x00bf0ba4 in AuxiliaryProcessMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
#5 0x00d3663b in ?? () from /usr/lib/postgresql/8.4/bin/postgres<br />
#6 0x00d39782 in ?? () from /usr/lib/postgresql/8.4/bin/postgres<br />
#7 <signal handler called><br />
#8 0x00a65422 in __kernel_vsyscall ()<br />
#9 0x00686a3d in ___newselect_nocancel () from /lib/tls/i686/cmov/libc.so.6<br />
#10 0x00d37bee in ?? () from /usr/lib/postgresql/8.4/bin/postgres<br />
#11 0x00d3afbe in PostmasterMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
#12 0x00cdc0dc in main () from /usr/lib/postgresql/8.4/bin/postgres<br />
</pre><br />
<br />
This example shows a stack trace that does not include function arguments. There may or may not be function arguments on your system, depending on obscure details largely outside your control, like whether or not Postgres was originally built to omit frame pointers, DWARF version, etc. In general, the situation with getting backtraces on mainstream Linux platforms has improved significantly since this example backtrace was originally added. These days, is often <b>better to use "bt full" instead of "bt"</b>, since this can provide even more information (the values of local/stack variables during the crash). In general, the more information that you can provide for debugging, the better.<br />
<br />
If you don't have proper symbols installed, specify the wrong executable to gdb or fail to specify an executable at all, you'll see a <b>useless</b> backtrace like this following one:<br />
<br />
<pre><br />
$ sudo -u postgres gdb -q -c /var/lib/postgresql/8.4/main/core.10780.sig6.1271644870s <br />
Core was generated by `postgres: wal writer process '.<br />
Program terminated with signal 6, Aborted.<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
(gdb) bt<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
#1 0x00686a3d in ?? ()<br />
#2 0x00d3d406 in ?? ()<br />
#3 0x00bf0ba4 in ?? ()<br />
#4 0x00d3663b in ?? ()<br />
#5 0x00d39782 in ?? ()<br />
#6 <signal handler called><br />
#7 0x00a65422 in __kernel_vsyscall ()<br />
#8 0x00686a3d in ?? ()<br />
#9 0x00d3afbe in ?? ()<br />
#10 0x00cdc0dc in ?? ()<br />
#11 0x005d7b56 in ?? ()<br />
#12 0x00b8fad1 in ?? ()<br />
<br />
</pre><br />
<br />
If you get something like that, don't bother sending it in. If you didn't just get the executable path wrong, you'll probably need to install debugging symbols for PostgreSQL (or even re-build PostgreSQL with debugging enabled) and try again.<br />
<br />
=== Tracing problems when creating a cluster ===<br />
<br />
If you're running into a crash while trying to create a database cluster using ''initdb'', that may leave behind a core dump that you can analyze with gdb as described above. This should be the case if there's an assertion failure for example. You will probably need to give the ''--no-clean'' option to ''initdb'' to keep it from deleting the new data directory and the core file along with it.<br />
<br />
Another technique for finding bootstrap-time bugs is to manually feed the bootstrapping commands into bootstrap mode or single-user mode, with a data directory left over from ''initdb --no-clean''. This can help if there has been no PANIC that leaves a core dump, but just a FATAL or ERROR, for example. It's easy to attach GDB to such a backend.<br />
<br />
Also, try creating the data directory using from unpatched master, then triggering the crash with the patched backend, rather than initdb.<br />
<br />
== Dumping a page image from within GDB ==<br />
<br />
It is sometimes useful to post a file containing a [https://www.postgresql.org/docs/current/storage-page-layout.html raw page image] when reporting a problem on a community mailing list. Both tables and indexes consist of 8KiB-sized blocks/pages, which can be thought of as the fundamental unit of data storage. This is particularly likely to be helpful when the integrity of the data is suspect, such as when an assertion fails due to a bug that corrupts data. GDB makes it easy to do this from either an interactive session (though core dumps may have [https://www.postgresql.org/message-id/20200210195659.vx6slnxmoymp5yyo%40alap3.anarazel.de issues with dumping shared memory]).<br />
<br />
Example:<br />
<br />
<pre><br />
Breakpoint 1, _bt_split (rel=0x7f555b6f3460, itup_key=0x55d03a745d40, buf=232, cbuf=0, firstright=366, newitemoff=216, newitemsz=16, newitem=0x55d03a745d18, newitemonleft=true) at nbtinsert.c:1205<br />
1205 {<br />
(gdb) n<br />
1215 Buffer sbuf = InvalidBuffer;<br />
(gdb)<br />
1216 Page spage = NULL;<br />
(gdb)<br />
1217 BTPageOpaque sopaque = NULL;<br />
(gdb)<br />
1227 int indnatts = IndexRelationGetNumberOfAttributes(rel);<br />
(gdb)<br />
1228 int indnkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);<br />
(gdb)<br />
1231 rbuf = _bt_getbuf(rel, P_NEW, BT_WRITE);<br />
(gdb)<br />
1244 origpage = BufferGetPage(buf);<br />
(gdb)<br />
1245 leftpage = PageGetTempPage(origpage);<br />
(gdb)<br />
1246 rightpage = BufferGetPage(rbuf);<br />
(gdb)<br />
1248 origpagenumber = BufferGetBlockNumber(buf);<br />
(gdb)<br />
1249 rightpagenumber = BufferGetBlockNumber(rbuf);<br />
(gdb) dump binary memory /tmp/dump_block.page origpage (origpage + 8192)<br />
</pre><br />
<br />
The contents of the page "origpage" are now dumped to the file "/tmp/dump_block.page", which will be precisely 8192 bytes in size. This works wherever the "Page" C type appears (which is actually a typedef defined in bufpage.h -- an unadorned "Page" is actually a char pointer). A "Page" variable is a raw pointer to a page image, typically the authoritative/current page stored in shared_buffers.<br />
<br />
=== pg_hexedit ===<br />
<br />
Note also that the Postgres hex editor tool [https://github.com/petergeoghegan/pg_hexedit pg_hexedit] can quickly [https://github.com/petergeoghegan/pg_hexedit#using-pg_hexedit-while-debugging-postgres-with-gdb visualize page images within GDB] with intuitive tags and annotations. It might be easier to use pg_hexedit when it isn't initially clear what page images are of interest, or when multiple images of the page from the same block need to be captured over time, as a test case is run.<br />
<br />
=== contrib/pageinspect page dump ===<br />
<br />
When it isn't convenient to use GDB, and when it isn't necessary to get a page image that is exactly current at the time of a crash, it is possible to dump an arbitrary page to a file in a more lightweight fashion using [https://www.postgresql.org/docs/current/pageinspect.html contrib/pageinspect]. For example, the following interactive shell session dumps the current page image in block 42 for the index 'pgbench_pkey':<br />
<br />
<pre><br />
$ psql -c "create extension pageinspect"<br />
CREATE EXTENSION<br />
$ psql -XAtc "SELECT encode(get_raw_page('pgbench_pkey', 42),'base64')" | base64 -d > dump_block_42.page<br />
</pre><br />
<br />
This assumes that it is possible to connect as a superuser using psql, and that the base64 program is in the user's $PATH. The GNU coreutils package generally includes base64, so it will already be available on most Linux installations. Note that it may be necessary to install an operating system package named "postgresql-contrib" or similar before the pageinspect extension will be available to install.<br />
<br />
Typically, the easiest way of following this procedure is to become the postgres operating system user first (e.g., through "su postgres").<br />
<br />
== Starting Postgres under GDB ==<br />
<br />
Debugging multi-process applications like PostgreSQL has historically been very painful with GDB. Thankfully with recent 7.x releases, this has been improved greatly by "inferiors" (GDB's term for multiple debugged processes).<br />
<br />
NB! This is still quite fragile, so don't expect to be able to do this in production.<br />
<br />
<source lang="bash"><br />
# Stop server<br />
pg_ctl -D /path/to/data stop -m fast<br />
# Launch postgres via gdb<br />
gdb --args postgres -D /path/to/data<br />
</source><br />
<br />
Now, in the GDB shell, use these commands to set up an environment:<br />
<br />
<source lang="bash"><br />
# We have scroll bars in the year 2012!<br />
set pagination off<br />
# Attach to both parent and child on fork<br />
set detach-on-fork off<br />
# Stop/resume all processes<br />
set schedule-multiple on<br />
<br />
# Usually don't care about these signals<br />
handle SIGUSR1 noprint nostop<br />
handle SIGUSR2 noprint nostop<br />
<br />
# Make GDB's expression evaluation work with most common Postgres Macros (works with Linux).<br />
# Per https://www.postgresql.org/message-id/20130731021434.GE19053@alap2.anarazel.de,<br />
# have many Postgres macros work if these are defined (useful for TOAST stuff,<br />
# varlena stuff, etc):<br />
macro define __builtin_offsetof(T, F) ((int) &(((T *) 0)->F))<br />
macro define __extension__<br />
<br />
# Ugly hack so we don't break on process exit<br />
python gdb.events.exited.connect(lambda x: [gdb.execute('inferior 1'), gdb.post_event(lambda: gdb.execute('continue'))])<br />
<br />
# Phew! Run it.<br />
run<br />
</source><br />
<br />
To get a list of processes, run <code>info inferior</code>. To switch to another process, run <code>inferior <i>NUM</i></code>.<br />
<br />
== Recording Postgres using rr Record and Replay Framework ==<br />
<br />
PostgreSQL 13 can be debugged using [https://rr-project.org the rr debugging recorder]. This section describes some useful workflows for using rr to debug Postgres. It is primarily written for Postgres hackers, though rr could also be used when reporting a bug.<br />
<br />
=== Version compatibility ===<br />
<br />
Commit {{PgCommitURL|fc3f4453a2bc95549682e23600b22e658cb2d6d7}} resolved an issue that made it hard to use rr with earlier Postgres versions, so there might be problems on those versions. Also, earlier versions of rr distributed with older/LTS Linux OS versions might not have support for syscalls that are used by Postgres, such as <code>sync_file_range()</code>. All of these issues probably have fairly straightforward workarounds (e.g. you could start Postgres with <code>--wal_writer_flush_after=0 --backend_flush_after=0 --bgwriter_flush_after=0 --checkpoint_flush_after=0</code>).<br />
<br />
=== Postgres settings ===<br />
<br />
A script that records a postgres session using rr might could consist of the following example snippet:<br />
<br />
<source lang="bash"><br />
rr record -M /code/postgresql/$BRANCH/install/bin/postgres \<br />
-D /code/postgresql/$BRANCH/data \<br />
--log_line_prefix="%m %p " \<br />
--effective_cache_size=1GB \<br />
--random_page_cost=4.0 \<br />
--work_mem=4MB \<br />
--maintenance_work_mem=64MB \<br />
--fsync=off \<br />
--log_statement=all \<br />
--log_min_messages=DEBUG5 \<br />
--max_connections=50 \<br />
--shared_buffers=32MB<br />
</source><br />
<br />
Most of the details here are somewhat arbitrary. The general idea is to make log output as verbose as possible, and to keep the amount of memory used by the server low.<br />
<br />
It is quite practical to run "make installcheck" against the server when Postgres is run with "rr record", recording the entire execution. This is not much slower than just running the tests against a regular debug build of Postgres. It's still much faster than Valgrind, for example. Replaying the recording seems to be where having a high end machine helps a lot.<br />
<br />
=== Event numbers in the log ===<br />
<br />
Once the tests are done, stop Postgres in the usual way (e.g. Ctrl + C). The recording is saved to the <code>$HOME/.local/share/rr/</code> directory on most Linux distros. rr creates a directory for each distinct recording in this parent directory. rr also maintains a symlink (<code>latest-trace</code>) that points to the latest recording directory, which is often used when replaying a recording. Be careful to avoid accidentally leaving too many recordings around. They can be rather large.<br />
<br />
The record/Postgres terminal has output that looks like this (when the example "rr record" recipe is used):<br />
<br />
<pre><br />
[rr 1786705 1241867]2020-04-04 21:55:05.018 PDT 1786705 DEBUG: CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 63992/1/2<br />
[rr 1786705 1241898]2020-04-04 21:55:05.019 PDT 1786705 DEBUG: StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0<br />
[rr 1786705 1241902]2020-04-04 21:55:05.019 PDT 1786705 LOG: statement: CREATE TYPE test_type_empty AS ();<br />
[rr 1786705 1241906]2020-04-04 21:55:05.020 PDT 1786705 DEBUG: CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 63993/1/1<br />
[rr 1786705 1241936]2020-04-04 21:55:05.020 PDT 1786705 DEBUG: StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0<br />
[rr 1786705 1241940]2020-04-04 21:55:05.020 PDT 1786705 LOG: statement: DROP TYPE test_type_empty;<br />
[rr 1786705 1241944]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: drop auto-cascades to composite type test_type_empty<br />
[rr 1786705 1241948]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: drop auto-cascades to type test_type_empty[]<br />
[rr 1786705 1241952]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: MultiXact: setting OldestMember[2] = 9<br />
[rr 1786705 1241956]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 63994/1/3<br />
</pre><br />
<br />
The part of each log line in square brackets comes from rr (since we used <code>-M</code> when recording) -- the first number is a PID, the second an event number. You probably won't care about the PIDs, though, since the event number alone unambiguously identifies a particular "event" in a particular backend (rr recordings are single threaded, even when there are multiple threads or processes). Suppose you want to get to the <code>CREATE TYPE test_type_empty AS ()</code> query -- you can get to the end of the query by replaying the recording with this option:<br />
<br />
<source lang="bash"><br />
$ rr replay -M -g 1241902<br />
</source><br />
<br />
Replaying the recording like this will take you to the point where the Postgres backend prints the log message at the end of executing the example query -- you will get a gdb debug server (rr implements a gdb backend), and interactive gdb session. This isn't precisely the point of execution that will be of interest to you, but it's close enough. You can easily set a breakpoint to the precise function you happen to be interested in, and then [https://sourceware.org/gdb/current/onlinedocs/gdb/Reverse-Execution.html <code>reverse-continue</code>] to get there by going backwards.<br />
<br />
You can also find the point where a particular backend starts by using the fork option instead. So for the PID 1786705, that would look like:<br />
<br />
<source lang="bash"><br />
$ rr replay -M -f 1786705<br />
</source><br />
<br />
(Don't try to use the similar <code>-p</code> option, since that starts a debug server when the pid has been <code>exec</code>'d.)<br />
<br />
Note that saving the output of a recording using standard tools like "tee" seems to have some issues [https://github.com/mozilla/rr/issues/91]. It may be helpful to get log output (complete with these event numbers) by doing an "autopilot" replay, like this:<br />
<br />
<source lang="bash"><br />
$ rr replay -M -a &> rr.log<br />
</source><br />
<br />
You now have a log file that can be searched for a good event number, as a starting point. This may be a practical necessity when running "make installcheck" or a custom test suite, since there might be megabytes of log output. You usually don't need to bother to generate logs in this way, though. It might take a few minutes to do an autopilot replay, since rr will replay everything that was recorded in sub-realtime.<br />
<br />
=== Jumping back and forth through a recording using GDB commands ===<br />
<br />
Once you have a rough idea of where and when a bug manifests itself in your rr recording, you'll need to actually debug the issue using gdb. Often the natural approach is to jump back and forth through the recording to track the issue down in whatever backend is known to be misbehaving.<br />
<br />
You can check the current event number once connected to gdb using gdb's "when" command, which can be useful when determining which point of execution you've reached relative to the high level output from "make check" (assuming the <code>-M</code> option was used to get event numbers there):<br />
<br />
<pre><br />
(rr) when<br />
Current event: 379377<br />
</pre><br />
<br />
Since event numbers are shared by processes/threads, which are alway executed serially during recording, event numbers are a generic way of reasoning about how far along the recording is, <b>within and across processes</b>. We are not limited to attaching our debugger to processes that happen to be Postgres backends.<br />
<br />
rr also supports gdb's <code>checkpoint</code>, <code>restart</code> and <code>delete</code> checkpoint commands; see [https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html#Checkpoint_002fRestart the relevant section of the GDB docs]. These are useful because they allow gdb to track interesting points in execution directly, at a finer granularity than "event number"; a new event number is created when there is a syscall, which might be far too coarse a granularity to be useful when actually zeroing in on a problem in one particular backend/process.<br />
<br />
=== Watchpoints and reverse execution ===<br />
<br />
Because rr supports reverse debugging, watchpoints are much more useful. Note that you should generally use <code>watch -l expr</code> rather than just using <code>watch expr</code>. Without -l, reverse execution is often very slow or apparently buggy, because gdb will try to reevaluate the expression as the program executes through different scopes.<br />
<br />
=== Debugging tap tests ===<br />
<br />
rr really shines when debugging things like tap tests, where there is complex scaffolding that may run multiple Postgres servers. You can run an entire "rr record make check", without having to worry about how that scaffolding works. Once you have useful PIDs (or event numbers) to work off of, it won't take too long to get an interactive debugging session in the backend of interest. You could get a PID for a backend of interest from the logs that appear in the <code>./tmp_check/log</code> directory once you're done with recording "make check" execution. From there, you can start "rr replay" by passing the relevant PID as the <code>-f</code> argument.<br />
<br />
Example replay of a "make check" session:<br />
<br />
<pre><br />
$ rr replay -M -f 2247718<br />
[rr 2246854 304]make -C ../../../src/backend generated-headers<br />
[rr 2246855 629]make[1]: Entering directory '/code/postgresql/patch/build/src/backend'<br />
[rr 2246855 631]make -C catalog distprep generated-header-symlinks<br />
[rr 2246856 984]make[2]: Entering directory '/code/postgresql/patch/build/src/backend/catalog'<br />
<br />
*** SNIP -- Remaining "make check" output omitted for brevity ***<br />
<br />
--------------------------------------------------<br />
---> Reached target process 2247718 at event 379377.<br />
--------------------------------------------------<br />
Reading symbols from /usr/bin/../lib/rr/librrpreload.so...<br />
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...<br />
Reading symbols from /usr/lib/debug/.build-id/0b/4031a3ab06ec61be1546960b4d1dad979d15ce.debug...<br />
<br />
*** SNIP ***<br />
<br />
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/libicudata.so.66)<br />
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...<br />
Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.31.so...<br />
0x0000000070000002 in ?? ()<br />
(rr) bt<br />
#0 0x0000000070000002 in ?? ()<br />
#1 0x00007f0d2c25c3b6 in _raw_syscall () at raw_syscall.S:120<br />
#2 0x00007f0d2c2582ff in traced_raw_syscall (call=call@entry=0x681fffa0) at syscallbuf.c:229<br />
#3 0x00007f0d2c259978 in sys_fcntl (call=<optimized out>) at syscallbuf.c:1291<br />
#4 syscall_hook_internal (call=0x681fffa0) at syscallbuf.c:2855<br />
#5 syscall_hook (call=0x681fffa0) at syscallbuf.c:2987<br />
#6 0x00007f0d2c2581da in _syscall_hook_trampoline () at syscall_hook.S:282<br />
#7 0x00007f0d2c25820a in __morestack () at syscall_hook.S:417<br />
#8 0x00007f0d2c258225 in _syscall_hook_trampoline_48_3d_00_f0_ff_ff () at syscall_hook.S:428<br />
#9 0x00007f0d2b5a9f15 in arch_fork (ctid=0x7f0d297bee50) at arch-fork.h:49<br />
#10 __libc_fork () at fork.c:76<br />
#11 0x00005620ae898e53 in fork_process () at fork_process.c:62<br />
#12 0x00005620ae8aab39 in BackendStartup (port=0x5620b0c1f600) at postmaster.c:4187<br />
#13 0x00005620ae8a6d29 in ServerLoop () at postmaster.c:1727<br />
#14 0x00005620ae8a64c2 in PostmasterMain (argc=4, argv=0x5620b0bf19e0) at postmaster.c:1400<br />
#15 0x00005620ae7a8247 in main (argc=4, argv=0x5620b0bf19e0) at main.c:210<br />
</pre><br />
<br />
=== Debugging race conditions ===<br />
<br />
rr can be used to [https://postgr.es/m/CAH2-WznTb6-0fjW4WPzNQh4mFvBH86J7bqZpNqteVUzo8p=6Hg@mail.gmail.com isolate hard to reproduce race condition bugs]. The single threaded nature of rr recording/execution seems to make it harder to reproduce bugs involving concurrent execution. However, using rr's [https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html chaos mode] option (by using the <code>-h</code> argument with rr record) seems to increase the odds of successfully reproducing a problem. It might still take a few attempts, but you only have to get lucky once.<br />
<br />
=== Packing a recording ===<br />
<br />
rr pack can be used to save a recording in a fairly stable format -- it copies the needed files into the trace:<br />
<br />
<source lang="bash"><br />
$ rr pack<br />
</source><br />
<br />
This could be useful if you wanted to save a recording for more than a day or two. Because every single detail of the recording (e.g. pointers, PIDs) is stable, you can treat a recording as a totally self contained thing.<br />
<br />
=== rr resources ===<br />
<br />
[https://github.com/mozilla/rr/wiki/Usage Usage - rr wiki]<br />
<br />
[https://github.com/mozilla/rr/wiki/Debugging-protips Debugging protips - rr wiki] <br />
<br />
<br />
[[Category:Operating system]]</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD&diff=37581Getting a stack trace of a running PostgreSQL backend on Linux/BSD2023-02-19T00:42:44Z<p>Pgeoghegan: Recommend the use of GDB checkpoints</p>
<hr />
<div>[[Generating a stack trace of a PostgreSQL backend|Up to parent]]<br />
<br />
== Linux and BSD ==<br />
<br />
Linux and BSD systems generally use the [http://gcc.gnu.org/ GNU compiler collection] and the [http://www.gnu.org/software/gdb/ GNU Debugger] ("gdb"). It's pretty trivial to get a stack trace of a process.<br />
<br />
(If you want more than just a stack trace, take a look at the [[Developer FAQ]] which covers interactive debugging).<br />
<br />
=== Installing External symbols ===<br />
<br />
(BSD users who installed from ports can skip this)<br />
<br />
On many Linux systems, debugging info is separated out from program binaries and stored separately. It's often not installed when you install a package, so if you want to debug the program (say, get a stack trace) you will need to install debug info packages. Unfortunately, the names of these packages vary depending on your distro, as does the procedure for installing them.<br />
<br />
Some generic instructions (unrelated to PostgreSQL) are maintained on the GNOME Wiki [http://live.gnome.org/GettingTraces/DistroSpecificInstructions here].<br />
<br />
==== On Debian ====<br />
<br />
http://wiki.debian.org/HowToGetABacktrace<br />
<br />
Debian Squeeze (6.x) users will also need to install gdb 7.3 from backports, as the gdb shipped in Squeeze doesn't understand the PIE executables used in newer PostgreSQL builds.<br />
<br />
==== On Ubuntu ====<br />
<br />
First, follow the instructions on the Ubuntu wiki entry [https://wiki.edubuntu.org/DebuggingProgramCrash DebuggingProgramCrash]. <br />
<br />
Once you've finished enabling the use of debug info packages as described, you will need to use the <code>list-dbgsym-packages.sh</code> script linked to on that wiki article to get a list of debug packages you need. Installing the debug package for postgresql alone is <i>not</i> sufficient. <br />
<br />
After following the instructions on the Ubuntu wiki, download the script to your desktop, open a terminal, and run:<br/><br />
<pre><br />
$ sudo apt-get install $(sudo bash Desktop/list-dbgsym-packages.sh -t -p $(pidof -s postgres))<br />
</pre><br />
<br />
==== On Fedora ====<br />
<br />
All Fedora versions: [https://fedoraproject.org/wiki/StackTraces#debuginfo FedoraProject.org - StackTraces]<br />
<br />
==== Other distros ====<br />
<br />
In general, you need to install at least the debug symbol packages for the PostgreSQL server and client as well as any common package that may exist, and the debug symbol package for libc. It's a good idea to add debug symbols for the other libraries PostgreSQL uses in case the problem you're having arises in or touches on one of those libraries.<br />
<br />
=== Collecting a stack trace ===<br />
<br />
==== How to tell if a stack trace is any good ====<br />
<br />
Read this section and keep it in mind as you collect information using the instructions below. Making sure the information you collect is actually useful will save you, and everybody else, time and hassle.<br />
<br />
It is vitally important to have debugging symbols available to get a useful stack trace. If you do not have the required symbols installed, backtraces will contain lots of entries like this:<br />
<br />
<pre><br />
#1 0x00686a3d in ?? ()<br />
#2 0x00d3d406 in ?? ()<br />
#3 0x00bf0ba4 in ?? ()<br />
#4 0x00d3663b in ?? ()<br />
#5 0x00d39782 in ?? ()<br />
</pre><br />
<br />
... which are completely useless for debugging without access to your system (and almost useless with access). If you see results like the above, you need to install debugging symbol packages, or even re-build postgresql with debugging enabled. <b>Do not bother collecting such backtraces, they are not useful.</b><br />
<br />
Sometimes you'll get backtraces that contain just the function name and the executable it's within, not source code file names and line numbers or parameters. Such output will have lines like this:<br />
<br />
<pre><br />
#11 0x00d3afbe in PostmasterMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
</pre><br />
<br />
This isn't ideal, but is a lot better than nothing. Installing debug information packages should give an even more detailed stack trace with line number and argument information, like this:<br />
<br />
<pre><br />
#9 0xb758d97e in PostmasterMain (argc=5, argv=0xb813a0e8) at postmaster.c:1040<br />
</pre><br />
<br />
... which is the most useful for tracking down your problem. Note the reference to a source file and line number instead of just an executable name.<br />
<br />
==== Identifying the backend to connect to ====<br />
<br />
You need to know the process ID of the postgresql backend to connect to. If you're interested in a backend that's using lots of CPU it might show up in <code>top</code>. If you have a current connection to the backend you're interested in, use <code>select pg_backend_pid()</code> to get its process ID. Otherwise, the <code>pg_catalog.pg_stat_activity</code> and/or <code>pg_catalog.pg_locks</code> views may be useful in identifying the backend of interest; see the "procpid" column in those views.<br />
<br />
==== Attaching gdb to the backend ====<br />
<br />
Once you know the process ID to connect to, run:<br />
<br />
<pre><br />
sudo gdb -p pid<br />
</pre><br />
<br />
where "pid" is the process ID of the backend. GDB will pause the execution of the process you specified and drop you into interactive mode (the <code>(gdb)</code> prompt) after showing the call the backend is currently running, eg:<br />
<br />
<pre><br />
0xb7c73424 in __kernel_vsyscall ()<br />
(gdb) <br />
</pre><br />
<br />
You'll want to tell gdb to save a log of the session to a file, so at the gdb prompt enter:<br />
<br />
<pre><br />
(gdb) set pagination off<br />
(gdb) set logging file debuglog.txt<br />
(gdb) set logging on<br />
</pre><br />
<br />
gdb is now saving all input and output to a file, <code>debuglog.txt</code>, in the directory in which you started gdb.<br />
<br />
At this point execution of the backend is still paused. It can even hold up other backends, so I recommend that you tell it to resume executing normally with the "cont" command:<br />
<br />
<pre><br />
(gdb) cont<br />
Continuing.<br />
</pre><br />
<br />
The backend is now running normally, as if gdb wasn't connected to it.<br />
<br />
==== Getting the trace ====<br />
<br />
OK, with gdb connected you're ready to get a useful stack trace.<br />
<br />
In addition to the instructions below, you can find some useful tips about using gdb with postgresql backends on the [[Developer_FAQ#What_debugging_features_are_available.3F|Developer FAQ]].<br />
<br />
==== Getting representative traces from a running backend ====<br />
<br />
If you're concerned with a case that's taking way too long to execute a query, is using too much CPU, or appears to be in an infinite loop, you'll want to <i>repeatedly</i> interrupt its execution, get a stack trace, and let it resume executing. Having a collection of several stack traces helps provide a better idea of where it's spending its time.<br />
<br />
You interrupt the backend and get back to the gdb command line with ^C (control-C). Once at the gdb command line, you use the "bt" command to get a backtrace, then the "cont" command to resume normal backend execution.<br />
<br />
Once you've collected a few backtraces, detach then exit gdb at the gdb interactive prompt:<br />
<br />
<pre><br />
(gdb) detach<br />
Detaching from program: /usr/lib/postgresql/8.3/bin/postgres, process 12912<br />
(gdb) quit<br />
user@host:~$<br />
</pre><br />
<br />
An alternative approach is to use the <code>gcore</code> program to save a series of core dumps of the running program without disrupting its execution. Those core dumps may then be examined at your leisure, giving you time to get more than just a backtrace because you're not holding up the backend's execution while you think and type.<br />
<br />
==== Getting a trace from the point of an error report ====<br />
<br />
If you are trying to find out the cause of an unexpected error, the most useful thing to do is to set a breakpoint at '''errfinish''' before you let the backend continue:<br />
<br />
<pre><br />
(gdb) b errfinish<br />
Breakpoint 1 at 0x80ced0: file elog.c, line 414.<br />
(gdb) cont<br />
Continuing.<br />
</pre><br />
<br />
Now, in your connected psql session, run whatever query is needed to provoke the error. When it happens, the backend will stop execution at '''errfinish'''.<br />
Collect your backtrace with '''bt''', then '''quit''' (or, possibly, '''cont''' if you want to do it again).<br />
<br />
A breakpoint at '''errfinish''' will capture generation of not only ERROR reports, but also NOTICE, LOG, and any other message that isn't suppressed by '''client_min_messages'''<br />
or '''log_min_messages'''. You may want to adjust those settings to avoid having to continue through a bunch of unrelated messages.<br />
<br />
==== Getting a trace from a reproducibly crashing backend ====<br />
<br />
GDB will automatically interrupt the execution of a program if it detects a crash. So, once you've attached gdb to the backend you expect to crash, you just let it continue execution as normal and do whatever you need to to make the backend crash.<br />
<br />
gdb will drop you into interactive mode as the backend crashes. At the <code>gdb</code> prompt you can enter the <code>bt</code> command to get a stack trace of the crash, then <code>cont</code> to continue execution. When gdb reports the process has exited, use the <code>quit</code> command.<br />
<br />
Alternately, you can collect a core file as explained below, but it's probably more hassle than it's worth if you know which backend to attach gdb to before it crashes.<br />
<br />
==== Getting a trace from a randomly crashing backend ====<br />
<br />
It's a lot harder to get a stack trace from a backend that's crashing when you don't know why it's crashing, what causes a backend to crash, or which backends will crash when. For this, you generally need to enable the generation of core files, which are debuggable dumps of a program's state that are generated by the operating system when the program crashes. <br />
<br />
===== Enabling core dumps =====<br />
<br />
[http://www.cyberciti.biz/tips/linux-core-dumps.html This article provides a useful primer on core dumps on Linux].<br />
<br />
On a Linux system you can check to see if core file generation is enabled for a process by examining /proc/$pid/limits, where $pid is the process ID of interest. "Max core file size" should be non-zero.<br />
<br />
Generally, adding "ulimit -c unlimited" to the top of the PostgreSQL startup script and restarting postgresql is sufficient to enable core dump collection. Make sure you have plenty of free space in your PostgreSQL data directory, because that's where the core dumps will be written and they can be fairly big due to Pg's use of shared memory. It may be useful to <b>temporarily reduce the size of shared_buffers</b> within postgresql.conf. This avoids core dumps that make the system unresponsive for minutes at a time, which can happen when shared_buffers is more than a few gigabytes. Reducing shared_buffers significantly will usually not make the server intolerably slow, since PostgreSQL will make increased use of the filesystem cache.<br />
<br />
On a Linux system it's also worth changing the file name format used for core dumps so that core dumps don't overwrite each other. The <code>/proc/sys/kernel/core_pattern</code> file controls this. I suggest <code>core.%p.sig%s.%ts</code>, which will record the process's PID, the signal that killed it, and the timestamp at which the core was generated. See <code>man 5 core</code>. To apply the settings change just run <code>echo core.%p.sig%s.%ts | sudo tee -a /proc/sys/kernel/core_pattern</code>.<br />
<br />
You can test whether core dumps are enabled by starting a `psql' session, finding the backend pid for it using the instructions given above, then killing it with "kill -ABRT pidofbackend" (where pidofbackend is the PID of the postgres backend, NOT the pid of psql). You should see a core file appear in your postgresql data directory.<br />
<br />
===== Debugging the core dump =====<br />
<br />
Once you've enabled core dumps, you need to wait until you see a backend crash. A core dump will be generated by the operating system, and you'll be able to attach gdb to it to collect a stack trace or other information. <br />
<br />
You need to tell gdb what executable file generated the core if you want to get useful backtraces and other debugging information. To do this, just specify the postgres executable path then the core file path when invoking gdb, as shown below. If you do not know the location of the postgres executable, you can get it by examining /proc/$pid/exe for a running postgres instance. For example:<br />
<br />
<pre><br />
$ for f in `pgrep postgres`; do ls -l /proc/$f/exe; done<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:30 /proc/10621/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11052/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11053/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11054/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
lrwxrwxrwx 1 postgres postgres 0 2010-04-19 10:51 /proc/11055/exe -> /usr/lib/postgresql/8.4/bin/postgres<br />
</pre><br />
<br />
... we can see from the above that the postgres executable on my (Ubuntu) system is <code>/usr/lib/postgresql/8.4/bin/postgres</code>.<br />
<br />
Once you know the executable path and the core file location, just run gdb with those as arguments, ie <code>gdb -q /path/to/postgres /path/to/core</code>. Now you can debug it as if it was a normal running postgres, as discussed in the sections above.<br />
<br />
===== Debugging the core dump - example =====<br />
<br />
For example, having just forced a postgres backend to crash with <code>kill -ABRT</code>, I have a core file named <code>core.10780.sig6.1271644870s</code> in <code>/var/lib/postgresql/8.4/main</code>, which is the data directory on my Ubuntu system. I've used /proc to find out that the executable for postgres on my system is <code>/usr/lib/postgresql/8.4/bin/postgres</code>.<br />
<br />
It's now easy to run GDB against it and request a backtrace:<br />
<br />
<pre><br />
$ sudo -u postgres gdb -q -c /var/lib/postgresql/8.4/main/core.10780.sig6.1271644870s /usr/lib/postgresql/8.4/bin/postgres<br />
Core was generated by `postgres: wal writer process '.<br />
Program terminated with signal 6, Aborted.<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
(gdb) bt<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
#1 0x00686a3d in ___newselect_nocancel () from /lib/tls/i686/cmov/libc.so.6<br />
#2 0x00e68d25 in pg_usleep () from /usr/lib/postgresql/8.4/bin/postgres<br />
#3 0x00d3d406 in WalWriterMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
#4 0x00bf0ba4 in AuxiliaryProcessMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
#5 0x00d3663b in ?? () from /usr/lib/postgresql/8.4/bin/postgres<br />
#6 0x00d39782 in ?? () from /usr/lib/postgresql/8.4/bin/postgres<br />
#7 <signal handler called><br />
#8 0x00a65422 in __kernel_vsyscall ()<br />
#9 0x00686a3d in ___newselect_nocancel () from /lib/tls/i686/cmov/libc.so.6<br />
#10 0x00d37bee in ?? () from /usr/lib/postgresql/8.4/bin/postgres<br />
#11 0x00d3afbe in PostmasterMain () from /usr/lib/postgresql/8.4/bin/postgres<br />
#12 0x00cdc0dc in main () from /usr/lib/postgresql/8.4/bin/postgres<br />
</pre><br />
<br />
This example shows a stack trace that does not include function arguments. There may or may not be function arguments on your system, depending on obscure details largely outside your control, like whether or not Postgres was originally built to omit frame pointers, DWARF version, etc. In general, the situation with getting backtraces on mainstream Linux platforms has improved significantly since this example backtrace was originally added. These days, is often <b>better to use "bt full" instead of "bt"</b>, since this can provide even more information (the values of local/stack variables during the crash). In general, the more information that you can provide for debugging, the better.<br />
<br />
If you don't have proper symbols installed, specify the wrong executable to gdb or fail to specify an executable at all, you'll see a <b>useless</b> backtrace like this following one:<br />
<br />
<pre><br />
$ sudo -u postgres gdb -q -c /var/lib/postgresql/8.4/main/core.10780.sig6.1271644870s <br />
Core was generated by `postgres: wal writer process '.<br />
Program terminated with signal 6, Aborted.<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
(gdb) bt<br />
#0 0x00a65422 in __kernel_vsyscall ()<br />
#1 0x00686a3d in ?? ()<br />
#2 0x00d3d406 in ?? ()<br />
#3 0x00bf0ba4 in ?? ()<br />
#4 0x00d3663b in ?? ()<br />
#5 0x00d39782 in ?? ()<br />
#6 <signal handler called><br />
#7 0x00a65422 in __kernel_vsyscall ()<br />
#8 0x00686a3d in ?? ()<br />
#9 0x00d3afbe in ?? ()<br />
#10 0x00cdc0dc in ?? ()<br />
#11 0x005d7b56 in ?? ()<br />
#12 0x00b8fad1 in ?? ()<br />
<br />
</pre><br />
<br />
If you get something like that, don't bother sending it in. If you didn't just get the executable path wrong, you'll probably need to install debugging symbols for PostgreSQL (or even re-build PostgreSQL with debugging enabled) and try again.<br />
<br />
=== Tracing problems when creating a cluster ===<br />
<br />
If you're running into a crash while trying to create a database cluster using ''initdb'', that may leave behind a core dump that you can analyze with gdb as described above. This should be the case if there's an assertion failure for example. You will probably need to give the ''--no-clean'' option to ''initdb'' to keep it from deleting the new data directory and the core file along with it.<br />
<br />
Another technique for finding bootstrap-time bugs is to manually feed the bootstrapping commands into bootstrap mode or single-user mode, with a data directory left over from ''initdb --no-clean''. This can help if there has been no PANIC that leaves a core dump, but just a FATAL or ERROR, for example. It's easy to attach GDB to such a backend.<br />
<br />
Also, try creating the data directory using from unpatched master, then triggering the crash with the patched backend, rather than initdb.<br />
<br />
== Dumping a page image from within GDB ==<br />
<br />
It is sometimes useful to post a file containing a [https://www.postgresql.org/docs/current/storage-page-layout.html raw page image] when reporting a problem on a community mailing list. Both tables and indexes consist of 8KiB-sized blocks/pages, which can be thought of as the fundamental unit of data storage. This is particularly likely to be helpful when the integrity of the data is suspect, such as when an assertion fails due to a bug that corrupts data. GDB makes it easy to do this from either an interactive session (though core dumps may have [https://www.postgresql.org/message-id/20200210195659.vx6slnxmoymp5yyo%40alap3.anarazel.de issues with dumping shared memory]).<br />
<br />
Example:<br />
<br />
<pre><br />
Breakpoint 1, _bt_split (rel=0x7f555b6f3460, itup_key=0x55d03a745d40, buf=232, cbuf=0, firstright=366, newitemoff=216, newitemsz=16, newitem=0x55d03a745d18, newitemonleft=true) at nbtinsert.c:1205<br />
1205 {<br />
(gdb) n<br />
1215 Buffer sbuf = InvalidBuffer;<br />
(gdb)<br />
1216 Page spage = NULL;<br />
(gdb)<br />
1217 BTPageOpaque sopaque = NULL;<br />
(gdb)<br />
1227 int indnatts = IndexRelationGetNumberOfAttributes(rel);<br />
(gdb)<br />
1228 int indnkeyatts = IndexRelationGetNumberOfKeyAttributes(rel);<br />
(gdb)<br />
1231 rbuf = _bt_getbuf(rel, P_NEW, BT_WRITE);<br />
(gdb)<br />
1244 origpage = BufferGetPage(buf);<br />
(gdb)<br />
1245 leftpage = PageGetTempPage(origpage);<br />
(gdb)<br />
1246 rightpage = BufferGetPage(rbuf);<br />
(gdb)<br />
1248 origpagenumber = BufferGetBlockNumber(buf);<br />
(gdb)<br />
1249 rightpagenumber = BufferGetBlockNumber(rbuf);<br />
(gdb) dump binary memory /tmp/dump_block.page origpage (origpage + 8192)<br />
</pre><br />
<br />
The contents of the page "origpage" are now dumped to the file "/tmp/dump_block.page", which will be precisely 8192 bytes in size. This works wherever the "Page" C type appears (which is actually a typedef defined in bufpage.h -- an unadorned "Page" is actually a char pointer). A "Page" variable is a raw pointer to a page image, typically the authoritative/current page stored in shared_buffers.<br />
<br />
=== pg_hexedit ===<br />
<br />
Note also that the Postgres hex editor tool [https://github.com/petergeoghegan/pg_hexedit pg_hexedit] can quickly [https://github.com/petergeoghegan/pg_hexedit#using-pg_hexedit-while-debugging-postgres-with-gdb visualize page images within GDB] with intuitive tags and annotations. It might be easier to use pg_hexedit when it isn't initially clear what page images are of interest, or when multiple images of the page from the same block need to be captured over time, as a test case is run.<br />
<br />
=== contrib/pageinspect page dump ===<br />
<br />
When it isn't convenient to use GDB, and when it isn't necessary to get a page image that is exactly current at the time of a crash, it is possible to dump an arbitrary page to a file in a more lightweight fashion using [https://www.postgresql.org/docs/current/pageinspect.html contrib/pageinspect]. For example, the following interactive shell session dumps the current page image in block 42 for the index 'pgbench_pkey':<br />
<br />
<pre><br />
$ psql -c "create extension pageinspect"<br />
CREATE EXTENSION<br />
$ psql -XAtc "SELECT encode(get_raw_page('pgbench_pkey', 42),'base64')" | base64 -d > dump_block_42.page<br />
</pre><br />
<br />
This assumes that it is possible to connect as a superuser using psql, and that the base64 program is in the user's $PATH. The GNU coreutils package generally includes base64, so it will already be available on most Linux installations. Note that it may be necessary to install an operating system package named "postgresql-contrib" or similar before the pageinspect extension will be available to install.<br />
<br />
Typically, the easiest way of following this procedure is to become the postgres operating system user first (e.g., through "su postgres").<br />
<br />
== Starting Postgres under GDB ==<br />
<br />
Debugging multi-process applications like PostgreSQL has historically been very painful with GDB. Thankfully with recent 7.x releases, this has been improved greatly by "inferiors" (GDB's term for multiple debugged processes).<br />
<br />
NB! This is still quite fragile, so don't expect to be able to do this in production.<br />
<br />
<source lang="bash"><br />
# Stop server<br />
pg_ctl -D /path/to/data stop -m fast<br />
# Launch postgres via gdb<br />
gdb --args postgres -D /path/to/data<br />
</source><br />
<br />
Now, in the GDB shell, use these commands to set up an environment:<br />
<br />
<source lang="bash"><br />
# We have scroll bars in the year 2012!<br />
set pagination off<br />
# Attach to both parent and child on fork<br />
set detach-on-fork off<br />
# Stop/resume all processes<br />
set schedule-multiple on<br />
<br />
# Usually don't care about these signals<br />
handle SIGUSR1 noprint nostop<br />
handle SIGUSR2 noprint nostop<br />
<br />
# Make GDB's expression evaluation work with most common Postgres Macros (works with Linux).<br />
# Per https://www.postgresql.org/message-id/20130731021434.GE19053@alap2.anarazel.de,<br />
# have many Postgres macros work if these are defined (useful for TOAST stuff,<br />
# varlena stuff, etc):<br />
macro define __builtin_offsetof(T, F) ((int) &(((T *) 0)->F))<br />
macro define __extension__<br />
<br />
# Ugly hack so we don't break on process exit<br />
python gdb.events.exited.connect(lambda x: [gdb.execute('inferior 1'), gdb.post_event(lambda: gdb.execute('continue'))])<br />
<br />
# Phew! Run it.<br />
run<br />
</source><br />
<br />
To get a list of processes, run <code>info inferior</code>. To switch to another process, run <code>inferior <i>NUM</i></code>.<br />
<br />
== Recording Postgres using rr Record and Replay Framework ==<br />
<br />
PostgreSQL 13 can be debugged using [https://rr-project.org the rr debugging recorder]. This section describes some useful workflows for using rr to debug Postgres. It is primarily written for Postgres hackers, though rr could also be used when reporting a bug.<br />
<br />
=== Version compatibility ===<br />
<br />
Commit {{PgCommitURL|fc3f4453a2bc95549682e23600b22e658cb2d6d7}} resolved an issue that made it hard to use rr with earlier Postgres versions, so there might be problems on those versions. Also, earlier versions of rr distributed with older/LTS Linux OS versions might not have support for syscalls that are used by Postgres, such as <code>sync_file_range()</code>. All of these issues probably have fairly straightforward workarounds (e.g. you could start Postgres with <code>--wal_writer_flush_after=0 --backend_flush_after=0 --bgwriter_flush_after=0 --checkpoint_flush_after=0</code>).<br />
<br />
=== Postgres settings ===<br />
<br />
A script that records a postgres session using rr might could consist of the following example snippet:<br />
<br />
<source lang="bash"><br />
rr record -M /code/postgresql/$BRANCH/install/bin/postgres \<br />
-D /code/postgresql/$BRANCH/data \<br />
--log_line_prefix="%m %p " \<br />
--effective_cache_size=1GB \<br />
--random_page_cost=4.0 \<br />
--work_mem=4MB \<br />
--maintenance_work_mem=64MB \<br />
--fsync=off \<br />
--log_statement=all \<br />
--log_min_messages=DEBUG5 \<br />
--max_connections=50 \<br />
--shared_buffers=32MB<br />
</source><br />
<br />
Most of the details here are somewhat arbitrary. The general idea is to make log output as verbose as possible, and to keep the amount of memory used by the server low.<br />
<br />
It is quite practical to run "make installcheck" against the server when Postgres is run with "rr record", recording the entire execution. This is not much slower than just running the tests against a regular debug build of Postgres. It's still much faster than Valgrind, for example. Replaying the recording seems to be where having a high end machine helps a lot.<br />
<br />
=== Event numbers in the log ===<br />
<br />
Once the tests are done, stop Postgres in the usual way (e.g. Ctrl + C). The recording is saved to the <code>$HOME/.local/share/rr/</code> directory on most Linux distros. rr creates a directory for each distinct recording in this parent directory. rr also maintains a symlink (<code>latest-trace</code>) that points to the latest recording directory, which is often used when replaying a recording. Be careful to avoid accidentally leaving too many recordings around. They can be rather large.<br />
<br />
The record/Postgres terminal has output that looks like this (when the example "rr record" recipe is used):<br />
<br />
<pre><br />
[rr 1786705 1241867]2020-04-04 21:55:05.018 PDT 1786705 DEBUG: CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 63992/1/2<br />
[rr 1786705 1241898]2020-04-04 21:55:05.019 PDT 1786705 DEBUG: StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0<br />
[rr 1786705 1241902]2020-04-04 21:55:05.019 PDT 1786705 LOG: statement: CREATE TYPE test_type_empty AS ();<br />
[rr 1786705 1241906]2020-04-04 21:55:05.020 PDT 1786705 DEBUG: CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 63993/1/1<br />
[rr 1786705 1241936]2020-04-04 21:55:05.020 PDT 1786705 DEBUG: StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0<br />
[rr 1786705 1241940]2020-04-04 21:55:05.020 PDT 1786705 LOG: statement: DROP TYPE test_type_empty;<br />
[rr 1786705 1241944]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: drop auto-cascades to composite type test_type_empty<br />
[rr 1786705 1241948]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: drop auto-cascades to type test_type_empty[]<br />
[rr 1786705 1241952]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: MultiXact: setting OldestMember[2] = 9<br />
[rr 1786705 1241956]2020-04-04 21:55:05.021 PDT 1786705 DEBUG: CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 63994/1/3<br />
</pre><br />
<br />
The part of each log line in square brackets comes from rr (since we used <code>-M</code> when recording) -- the first number is a PID, the second an event number. You probably won't care about the PIDs, though, since the event number alone unambiguously identifies a particular "event" in a particular backend (rr recordings are single threaded, even when there are multiple threads or processes). Suppose you want to get to the <code>CREATE TYPE test_type_empty AS ()</code> query -- you can get to the end of the query by replaying the recording with this option:<br />
<br />
<source lang="bash"><br />
$ rr replay -M -g 1241902<br />
</source><br />
<br />
Replaying the recording like this will take you to the point where the Postgres backend prints the log message at the end of executing the example query -- you will get a gdb debug server (rr implements a gdb backend), and interactive gdb session. This isn't precisely the point of execution that will be of interest to you, but it's close enough. You can easily set a breakpoint to the precise function you happen to be interested in, and then [https://sourceware.org/gdb/current/onlinedocs/gdb/Reverse-Execution.html <code>reverse-continue</code>] to get there by going backwards.<br />
<br />
You can also find the point where a particular backend starts by using the fork option instead. So for the PID 1786705, that would look like:<br />
<br />
<source lang="bash"><br />
$ rr replay -M -f 1786705<br />
</source><br />
<br />
(Don't try to use the similar <code>-p</code> option, since that starts a debug server when the pid has been <code>exec</code>'d.)<br />
<br />
Note that saving the output of a recording using standard tools like "tee" seems to have some issues [https://github.com/mozilla/rr/issues/91]. It may be helpful to get log output (complete with these event numbers) by doing an "autopilot" replay, like this:<br />
<br />
<source lang="bash"><br />
$ rr replay -M -a &> rr.log<br />
</source><br />
<br />
You now have a log file that can be searched for a good event number, as a starting point. This may be a practical necessity when running "make installcheck" or a custom test suite, since there might be megabytes of log output. You usually don't need to bother to generate logs in this way, though. It might take a few minutes to do an autopilot replay, since rr will replay everything that was recorded in sub-realtime.<br />
<br />
=== Jumping back and forth through a recording use GDB commands ===<br />
<br />
Once you have a rough idea of where and when a bug manifests itself in your rr recording, you'll need to actually debug the issue using gdb. Often the natural approach is to jump back and forth through the recording to track the issue down in whatever backend is known to be misbehaving.<br />
<br />
You can check the current event number once connected to gdb using gdb's "when" command, which can be useful when determining which point of execution you've reached relative to the high level output from "make check" (assuming the <code>-M</code> option was used to get event numbers there):<br />
<br />
<pre><br />
(rr) when<br />
Current event: 379377<br />
</pre><br />
<br />
Since event numbers are shared by processes/threads, which are alway executed serially during recording, event numbers are a generic way of reasoning about how far along the recording is, <b>within and across processes</b>. We are not limited to attaching our debugger to processes that happen to be Postgres backends.<br />
<br />
rr also supports gdb's <code>checkpoint</code>, <code>restart</code> and <code>delete</code> checkpoint commands; see [https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html#Checkpoint_002fRestart the relevant section of the GDB docs]. These are useful because they allow gdb to track interesting points in execution directly, at a finer granularity than "event number"; a new event number is created when there is a syscall, which might be far too coarse a granularity to be useful when actually zeroing in on a problem in one particular backend/process.<br />
<br />
=== Watchpoints and reverse execution ===<br />
<br />
Because rr supports reverse debugging, watchpoints are much more useful. Note that you should generally use <code>watch -l expr</code> rather than just using <code>watch expr</code>. Without -l, reverse execution is often very slow or apparently buggy, because gdb will try to reevaluate the expression as the program executes through different scopes.<br />
<br />
=== Debugging tap tests ===<br />
<br />
rr really shines when debugging things like tap tests, where there is complex scaffolding that may run multiple Postgres servers. You can run an entire "rr record make check", without having to worry about how that scaffolding works. Once you have useful PIDs (or event numbers) to work off of, it won't take too long to get an interactive debugging session in the backend of interest. You could get a PID for a backend of interest from the logs that appear in the <code>./tmp_check/log</code> directory once you're done with recording "make check" execution. From there, you can start "rr replay" by passing the relevant PID as the <code>-f</code> argument.<br />
<br />
Example replay of a "make check" session:<br />
<br />
<pre><br />
$ rr replay -M -f 2247718<br />
[rr 2246854 304]make -C ../../../src/backend generated-headers<br />
[rr 2246855 629]make[1]: Entering directory '/code/postgresql/patch/build/src/backend'<br />
[rr 2246855 631]make -C catalog distprep generated-header-symlinks<br />
[rr 2246856 984]make[2]: Entering directory '/code/postgresql/patch/build/src/backend/catalog'<br />
<br />
*** SNIP -- Remaining "make check" output omitted for brevity ***<br />
<br />
--------------------------------------------------<br />
---> Reached target process 2247718 at event 379377.<br />
--------------------------------------------------<br />
Reading symbols from /usr/bin/../lib/rr/librrpreload.so...<br />
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...<br />
Reading symbols from /usr/lib/debug/.build-id/0b/4031a3ab06ec61be1546960b4d1dad979d15ce.debug...<br />
<br />
*** SNIP ***<br />
<br />
(No debugging symbols found in /usr/lib/x86_64-linux-gnu/libicudata.so.66)<br />
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...<br />
Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.31.so...<br />
0x0000000070000002 in ?? ()<br />
(rr) bt<br />
#0 0x0000000070000002 in ?? ()<br />
#1 0x00007f0d2c25c3b6 in _raw_syscall () at raw_syscall.S:120<br />
#2 0x00007f0d2c2582ff in traced_raw_syscall (call=call@entry=0x681fffa0) at syscallbuf.c:229<br />
#3 0x00007f0d2c259978 in sys_fcntl (call=<optimized out>) at syscallbuf.c:1291<br />
#4 syscall_hook_internal (call=0x681fffa0) at syscallbuf.c:2855<br />
#5 syscall_hook (call=0x681fffa0) at syscallbuf.c:2987<br />
#6 0x00007f0d2c2581da in _syscall_hook_trampoline () at syscall_hook.S:282<br />
#7 0x00007f0d2c25820a in __morestack () at syscall_hook.S:417<br />
#8 0x00007f0d2c258225 in _syscall_hook_trampoline_48_3d_00_f0_ff_ff () at syscall_hook.S:428<br />
#9 0x00007f0d2b5a9f15 in arch_fork (ctid=0x7f0d297bee50) at arch-fork.h:49<br />
#10 __libc_fork () at fork.c:76<br />
#11 0x00005620ae898e53 in fork_process () at fork_process.c:62<br />
#12 0x00005620ae8aab39 in BackendStartup (port=0x5620b0c1f600) at postmaster.c:4187<br />
#13 0x00005620ae8a6d29 in ServerLoop () at postmaster.c:1727<br />
#14 0x00005620ae8a64c2 in PostmasterMain (argc=4, argv=0x5620b0bf19e0) at postmaster.c:1400<br />
#15 0x00005620ae7a8247 in main (argc=4, argv=0x5620b0bf19e0) at main.c:210<br />
</pre><br />
<br />
=== Debugging race conditions ===<br />
<br />
rr can be used to [https://postgr.es/m/CAH2-WznTb6-0fjW4WPzNQh4mFvBH86J7bqZpNqteVUzo8p=6Hg@mail.gmail.com isolate hard to reproduce race condition bugs]. The single threaded nature of rr recording/execution seems to make it harder to reproduce bugs involving concurrent execution. However, using rr's [https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html chaos mode] option (by using the <code>-h</code> argument with rr record) seems to increase the odds of successfully reproducing a problem. It might still take a few attempts, but you only have to get lucky once.<br />
<br />
=== Packing a recording ===<br />
<br />
rr pack can be used to save a recording in a fairly stable format -- it copies the needed files into the trace:<br />
<br />
<source lang="bash"><br />
$ rr pack<br />
</source><br />
<br />
This could be useful if you wanted to save a recording for more than a day or two. Because every single detail of the recording (e.g. pointers, PIDs) is stable, you can treat a recording as a totally self contained thing.<br />
<br />
=== rr resources ===<br />
<br />
[https://github.com/mozilla/rr/wiki/Usage Usage - rr wiki]<br />
<br />
[https://github.com/mozilla/rr/wiki/Debugging-protips Debugging protips - rr wiki] <br />
<br />
<br />
[[Category:Operating system]]</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37552Meson2023-02-10T00:01:43Z<p>Pgeoghegan: /* Test structure */</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
|| "--suite setup" required to get a tmp_install directory; see below<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite setup --suite amcheck<br />
|| "--suite setup" required to get a tmp_install directory; see below<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
When running a specific test suite against a temporary throw away installation, <code>--suite setup</code> should generally be specified. Otherwise the tests could end up running against a stale <code>tmp_install</code> directory, causing general confusion. This [https://postgr.es/m/20230209205605.zo5gfhli22g2kdm2@awork3.anarazel.de workaround] is not required when running tests against an existing server (via the <code>running</code> test setup and variant test suites), since of course the installation directory being tested is whatever directory the external server installation uses.<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be omitted] when using a reasonably recent meson version (meson 0.46 or later), which we assume here.<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37551Meson2023-02-09T23:53:24Z<p>Pgeoghegan: /* Test structure */</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
|| "--suite setup" required to get a tmp_install directory; see below<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite setup --suite amcheck<br />
|| "--suite setup" required to get a tmp_install directory; see below<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
When running a specific test suite against a temporary throw away installation, <code>--suite setup</code> should generally be specified. Otherwise the tests could end up running against a stale <code>tmp_install</code> directory, causing general confusion. This [https://postgr.es/m/20230209205605.zo5gfhli22g2kdm2@awork3.anarazel.de workaround] is not required when running tests against an existing server (via the <code>running</code> test setup and variant test suites), since of course the installation directory being tested is whatever directory the external server installation uses.<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be ommitted] when using a reasonably recent meson version (meson 0.46 or later), which we assume here.<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37550Meson2023-02-09T23:51:07Z<p>Pgeoghegan: Explain "--suite setup" issue</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
|| "--suite setup" required to get a tmp_install directory; see below<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite setup --suite amcheck<br />
|| "--suite setup" required to get a tmp_install directory; see below<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
When running a specific test suite against a temporary throw away installation, <code>--suite setup</code> should generally be specified. Otherwise the tests could end up running against a stale <code>tmp_install</code> directory, causing general confusion. This workaround is not required when running tests against an existing server (via the <code>running</code> test setup and variant test suites), since of course the installation directory being tested is whatever directory the external server installation uses.<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be ommitted] when using a reasonably recent meson version (meson 0.46 or later), which we assume here.<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37549Meson2023-02-09T23:24:44Z<p>Pgeoghegan: Add missing "--suite setup" to test mappings, don't bother spelling out "postgresql:" top-level project name</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite setup --suite amcheck<br />
||<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be ommitted] when using a reasonably recent meson version (meson 0.46 or later), which we assume here.<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37548Meson2023-02-09T20:19:27Z<p>Pgeoghegan: /* Test structure */</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite postgresql:regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite postgresql:amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be ommitted] when using a reasonably recent meson version (meson 0.46 or later). For example, a simple <code>meson test -v --suite amcheck</code> will work (the <code>postgresql:</code> prefix is not truly required).<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
Note that there are distinct <code>running</code>/installcheck suites for most of the standard setup suites, though not all of the tests actually carry over to the <code>running</code> variant suites, as shown here:<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck-running<br />
ninja: no work to do.<br />
postgresql:amcheck-running / amcheck-running/regress<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37547Meson2023-02-09T20:11:58Z<p>Pgeoghegan: Reorder command translation sections a little, so we talk about basic build/setup options first, then talk about testing related options</p>
<hr />
<div>== PostgreSQL devel documentation ==<br />
<br />
See [https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section] of PostgreSQL devel docs.<br />
<br />
== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
==== Build directory ====<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite postgresql:regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite postgresql:amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be ommitted] when using a reasonably recent meson version (meson 0.46 or later). For example, a simple <code>meson test -v --suite amcheck</code> will work (the <code>postgresql:</code> prefix is not truly required).<br />
<br />
You can list all of the tests from a given suite as follows:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ meson test --list --suite amcheck<br />
ninja: no work to do.<br />
postgresql:amcheck / amcheck/regress<br />
postgresql:amcheck / amcheck/001_verify_heapam<br />
postgresql:amcheck / amcheck/002_cic<br />
postgresql:amcheck / amcheck/003_cic_2pc<br />
</pre><br />
<br />
==== Running individual regression test scripts via an installcheck-tests style workflow ====<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37546Meson2023-02-09T20:01:34Z<p>Pgeoghegan: /* Test related commands */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite postgresql:regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite postgresql:amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
==== Test structure ====<br />
<br />
Note that the top-level/default project name is <code>postgresql</code>, which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be ommitted] when using a reasonably recent meson version (meson 0.46 or later). For example, a simple <code>meson test -v --suite amcheck</code> will work (the <code>postgresql:</code> prefix is not truly required).<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37545Meson2023-02-09T19:51:23Z<p>Pgeoghegan: /* Test related commands */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite postgresql:regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite postgresql:amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
Note that the top-level/default project name is "postgresql", which is the only one we use in practice. The project name [https://mesonbuild.com/Unit-tests.html#run-subsets-of-tests can be ommitted] when using a reasonably recent meson version (meson 0.46 or later). For example, a simple "meson test -v --suite amcheck" will work (the "postgresql:" prefix is not truly required).<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37544Meson2023-02-09T18:50:29Z<p>Pgeoghegan: Prefer using --suite in meson test examples, use -v consistently to match autoconf's more verbose default behavior</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test -v<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test -v --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test -v --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific contrib test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running --suite postgresql:regress-running<br />
||<br />
|-<br />
|| run specific contrib test suite against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running --suite postgresql:amcheck-running<br />
|| "running" amcheck suite variant doesn't include TAP tests<br />
|}<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37519Meson2023-02-07T20:25:50Z<p>Pgeoghegan: /* Test related commands */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| "running" [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup test setup] is used to run tests against an existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37518Meson2023-02-07T03:33:26Z<p>Pgeoghegan: Add entry for meson install --only-changed</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| install files that changed only<br />
||<br />
|| meson install --only-changed<br />
|| Routinely shaves a few hundred milliseconds off install time<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup Setup] used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37517Meson2023-02-07T02:34:15Z<p>Pgeoghegan: /* Test related commands */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup Setup] used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test -v --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37516Meson2023-02-07T02:32:15Z<p>Pgeoghegan: /* Running individual regression test scripts via an installcheck-tests style workflow */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup Setup] used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37515Meson2023-02-07T02:26:42Z<p>Pgeoghegan: Add info about working around the lack of a installcheck-tests target with meson</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup Setup] used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== Running individual regression test scripts via an installcheck-tests style workflow ===<br />
<br />
The Postgres autoconf build system supports running a subset of regression test scripts against an existing server using the installcheck-tests target, as shown here:<br />
<br />
<pre><br />
/path/to/postgresql/build_autoconf $ make installcheck-tests TESTS="test_setup create_index"<br />
*** SNIP ***<br />
============== dropping database "regression" ==============<br />
SET<br />
DROP DATABASE<br />
============== creating database "regression" ==============<br />
CREATE DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
ALTER DATABASE<br />
============== running regression test queries ==============<br />
test test_setup ... ok 300 ms<br />
test create_index ... ok 1775 ms<br />
<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
You can work around the current lack of an equivalent meson facility by invoking pg_regress directly:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/regress/pg_regress --inputdir ../source/src/test/regress/ --dlpath=src/test/regress/ test_setup create_index<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
The same approach will also work for isolation tests:<br />
<br />
<pre><br />
/path/to/postgresql/build_meson $ src/test/isolation/pg_isolation_regress --inputdir ../source/src/test/isolation freeze-the-dead vacuum-no-cleanup-lock<br />
*** SNIP ***<br />
=====================<br />
All 2 tests passed.<br />
=====================<br />
</pre><br />
<br />
Note that this assumes that the meson build directory is 'build_meson', and that the Postgres source code directory is 'source'. The 'source' directory is located in the same directory as 'build_meson' in this example, (a directory layout often used with VPATH builds).<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37514Meson2023-02-07T01:41:25Z<p>Pgeoghegan: /* Test related commands */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| [https://mesonbuild.com/Reference-manual_functions.html#add_test_setup Setup] used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37513Meson2023-02-07T01:13:48Z<p>Pgeoghegan: /* Test related commands */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| Used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37512Meson2023-02-07T01:02:23Z<p>Pgeoghegan: Use specific suite (not test) in "make check" command mapping example</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| Used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific regression test suite<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v --suite postgresql:amcheck<br />
||<br />
|-<br />
|| run specific regression test<br />
||<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| Doesn't run TAP tests, unlike suite-level recipe<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
||<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37511Meson2023-02-07T00:41:23Z<p>Pgeoghegan: Reorder testing command translation items for clarity</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| list running/installcheck test variants<br />
||<br />
|| meson test --setup running --list<br />
|| Used when running specific tests against existing server<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific regression test<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| meson won't run TAP tests here (there are distinct non-regress tests for those)<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
|| meson won't run TAP tests here<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37510Meson2023-02-07T00:20:57Z<p>Pgeoghegan: /* Test related commands translation */</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
|| Test names cannot be directly used for '--setup running' (AKA 'make installcheck') testing<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
|| Limited to tests that support running against existing server<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific regression test<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| meson won't run TAP tests here (there are distinct non-regress tests for those)<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
|| Use 'meson test -v --setup running' to get spelling of any other "running" test<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37509Meson2023-02-07T00:19:27Z<p>Pgeoghegan: Split autoconf:meson translation table into two tables to improve readability</p>
<hr />
<div>== Autoconf:meson command translations ==<br />
<br />
=== Setup and build commands ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages<br />
|}<br />
<br />
=== Test related commands translation ===<br />
<br />
{|class="wikitable" style="margin:auto"<br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
|| Test names cannot be directly used for '--setup running' (AKA 'make installcheck') testing<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
|| Limited to tests that support running against existing server<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific regression test<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| meson won't run TAP tests here (there are distinct non-regress tests for those)<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
|| Use 'meson test -v --setup running' to get spelling of any other "running" test<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37508Meson2023-02-07T00:04:34Z<p>Pgeoghegan: Reorder a couple of items from table for clarity</p>
<hr />
<div>== Quickstart ==<br />
<br />
{|class="wikitable" style="margin:auto"<br />
|+ translation of configure, make, etc commands <br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages <br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
||<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
|| Limited to tests that support running against existing server<br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific regression test<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| meson won't run TAP tests here (there are distinct non-regress tests for those)<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
|| Use 'meson test -v --setup running' to get spelling of any other "running" test<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37507Meson2023-02-06T23:56:36Z<p>Pgeoghegan: Add "run specific regression test" example</p>
<hr />
<div>== Quickstart ==<br />
<br />
{|class="wikitable" style="margin:auto"<br />
|+ translation of configure, make, etc commands <br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
|| Limited to tests that support running against existing server<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages <br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run specific regression test<br />
|| cd contrib/amcheck; make check;<br />
|| meson test -v postgresql:amcheck / amcheck/regress<br />
|| meson won't run TAP tests here (there are distinct non-regress tests for those)<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib test against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
|| Use 'meson test -v --setup running' to get spelling of any other "running" test<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37506Meson2023-02-06T23:41:36Z<p>Pgeoghegan: Add "run specific contrib tests against existing server" example</p>
<hr />
<div>== Quickstart ==<br />
<br />
{|class="wikitable" style="margin:auto"<br />
|+ translation of configure, make, etc commands <br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
|| Limited to tests that support running against existing server<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages <br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| run specific contrib tests against existing server<br />
|| cd contrib/amcheck; make installcheck;<br />
|| meson test -v --setup running postgresql:amcheck-running / amcheck-running/regress<br />
|| Use 'meson test -v --setup running' to get spelling of any other "running" test<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== FreeBSD ===<br />
<br />
<pre><br />
pkg install meson ninja<br />
</pre><br />
<br />
Arguments to meson setup/configure to find ports libraries:<br />
<pre><br />
meson setup -Dextra_lib_dirs=/opt/local/lib -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Sample_Databases&diff=37503Sample Databases2023-02-04T22:46:31Z<p>Pgeoghegan: Add new link to conveniently available IMDB database custom format pg_dump</p>
<hr />
<div>Many database systems provide sample databases with the product. A good intro to popular ones that includes discussion of samples available for other databases is [http://www.barik.net/archive/2006/03/28/195425/ Sample Databases for PostgreSQL and More] (2006).<br />
<br />
One trivial sample that PostgreSQL ships with is the [[Pgbench]]. This has the advantage of being built-in and supporting a scalable data generator.<br />
<br />
* MySQL has a popular sample database named [https://dev.mysql.com/doc/sakila/en/ Sakila]. Sakila [https://github.com/jOOQ/jOOQ/tree/master/jOOQ-examples/Sakila has been ported to many databases] including Postgres.<br />
<br />
* [https://github.com/devrimgunduz/pagila Pagila] is a more idiomatic Postgres port of Sakila.<br />
<br />
* PgFoundry had [https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/ a collection of Postgres-compatible sample databases] but it has not been updated since 2008.<br />
* [https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/2QYZBT IMDB Data for JOB Workload], as used in the paper [https://doi.org/10.14778/2850583.2850594 "How Good are Query Optimizers, Really?"]. This data was generated using a [https://github.com/RyanMarcus/imdb_pg_dataset tool that is freely available on Github]. It is conveniently available to download as a <code>pg_dump -Fc</code> dump.<br />
* [http://www.imdb.com/interfaces#plain IMDB] - the original IMDB source data.<br />
* [https://theodi.org/blog/the-status-of-csvs-on-datagovuk The land registry file] from http://data.gov.uk has details of land sales in the UK, going back several decades, and is 3.5GB as of August 2016 (this applies only to the "complete" file, "pp-complete.csv"). No registration required.<br />
<pre><br />
-- Download file "pp-complete.csv", which has all records.<br />
-- If schema changes/field added, consult: https://www.gov.uk/guidance/about-the-price-paid-data<br />
<br />
-- Create table:<br />
CREATE TABLE land_registry_price_paid_uk(<br />
transaction uuid,<br />
price numeric,<br />
transfer_date date,<br />
postcode text,<br />
property_type char(1),<br />
newly_built boolean,<br />
duration char(1),<br />
paon text,<br />
saon text,<br />
street text,<br />
locality text,<br />
city text,<br />
district text,<br />
county text,<br />
ppd_category_type char(1),<br />
record_status char(1));<br />
<br />
-- Copy CSV data, with appropriate munging:<br />
COPY land_registry_price_paid_uk FROM '/path/to/pp-complete.csv' with (format csv, encoding 'win1252', header false, null '', quote '"', force_null (postcode, saon, paon, street, locality, city, district));<br />
</pre><br />
* [https://github.com/lorint/AdventureWorks-for-Postgres AdventureWorks 2014 for Postgres] - Scripts to set up the OLTP part of the go-to database used in training classes and for sample apps on the Microsoft stack. The result is 68 tables containing HR, sales, product, and purchasing data organized across 5 schemas. It represents a fictitious bicycle parts wholesaler with a hierarchy of nearly 300 employees, 500 products, 20000 customers, and 31000 sales each having an average of 4 line items. So it's big enough to be interesting, but not unwieldy. In addition to being a well-rounded OLTP sample, it is also a good choice to demonstrate ETL into a data warehouse. The code in some of the views demonstrates effective techniques for querying XML data.<br />
* [http://www.informatics.jax.org/downloads/database_backups/ Mouse Genome sample data set]. See [http://www.informatics.jax.org/software.shtml instructions]. Custom format dump, 1.9GB compressed, but restored database is tens of GB in size. MGI is the international database resource for the laboratory mouse, providing integrated genetic, genomic, and biological data to facilitate the study of human health and disease. MGI use PostgreSQL in production [http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3245042/], providing direct protocol access to researchers, so the custom format dump is not an afterthought. Apparently updated frequently.<br />
* Benchmarking databases such as [[DBT-2]] or [[TPC-H]] can be used as samples.<br />
* [http://www.freebase.com/docs/data_dumps Freebase] - Various wiki style data on places/people/things - ~600MB compressed<br />
* [https://www.omdb.org/content/About OMDB] - Open Media database, ~30MB compressed, 300MB when loaded - https://github.com/credativ/omdb-postgresql<br />
* [http://www.data.gov/ Data.gov] - US federal government data collection, see also [http://www.sunlightlabs.com/ Sunlight Labs]<br />
* [http://wiki.dbpedia.org/Downloads DBpedia] - Wikipedia data export project<br />
* [http://www.eoddata.com/ eoddata] - historic stock market data (requires registration - licence?)<br />
* [http://www.transtats.bts.gov/Tables.asp?DB_ID=120&DB_Name=Airline%20On-Time%20Performance%20Data&DB_Short_Name=On-Time RITA] - Airline On-Time Performance Data<br />
* [http://wiki.openstreetmap.org/wiki/Planet.osm Openstreetmap] - Openstreetmap source data<br />
* [https://ftp.ncbi.nih.gov/gene/DATA/ NCBI] - biological annotation from NCBI's ENTREZ system (updated daily)<br />
* [https://postgrespro.com/education/demodb Airlines Demo Database] - Airlines Demo Database provides database schema with several tables and meaningful content, which can be used for learning SQL and writing applications<br />
* [https://archive.org/details/stackexchange Stack Exchange Data Dump] - Anonymized dump of all user-contributed content on the Stack Exchange network (Stack Overflow, Server Fault...) under cc-by-sa 3.0 license. Use this tool to import XML dumps in PostgresQL : https://github.com/Networks-Learning/stackexchange-dump-to-postgres<br />
* [https://github.com/MuseumofModernArt/collection The Museum of Modern Art (MoMA) collection data] - This research dataset contains more than 130,000 records, representing all of the works that have been accessioned into MoMA’s collection and cataloged in our database. It includes basic metadata for each work, including title, artist, date made, medium, dimensions, and date acquired by the Museum. At this time, both datasets are available in CSV and JSON format, encoded in UTF-8. <br />
<br />
[[Category:Howto]]</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37421Meson2022-12-27T23:54:03Z<p>Pgeoghegan: Add "meson test --setup running" entry to quickstart table</p>
<hr />
<div>== Quickstart ==<br />
<br />
{|class="wikitable" style="margin:auto"<br />
|+ translation of configure, make, etc commands <br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| run all tests against existing server<br />
|| make installcheck-world<br />
|| meson test --setup running<br />
|| Limited to tests that support running against existing server<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages <br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running regress-running/regress<br />
||<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Meson&diff=37420Meson2022-12-27T23:46:18Z<p>Pgeoghegan: Add "meson test --setup running" entry to quickstart table</p>
<hr />
<div>== Quickstart ==<br />
<br />
{|class="wikitable" style="margin:auto"<br />
|+ translation of configure, make, etc commands <br />
!description<br />
!old command<br />
!new command<br />
!comment<br />
|-<br />
|| set up build tree<br />
|| ./configure [<options>]<br />
|| meson setup [<options>] [<build dir>] <source-dir> <br />
|| meson only supports building out of tree<br />
|-<br />
|| set up build tree for visual studio<br />
|| perl src/tools/msvc/mkvcbuild.pl<br />
|| meson setup --backend vs [<options>] [<build dir>] <source-dir><br />
|| configures build tree for one build type (debug or release or ...)<br />
|-<br />
|| show configure options<br />
|| ./configure --help<br />
|| meson configure<br />
|| shows options built into meson and PostgreSQL specific options<br />
|-<br />
|| set configure options<br />
|| ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br />
|| meson setup|configure -D$option=$value<br />
|| options can be set when setting up build tree (setup) and in existing build tree (configure)<br />
|- <br />
|| enable cassert<br />
|| --enable-cassert<br />
|| -Dcassert=true<br />
|-<br />
|| enable debug symbols<br />
|| ./configure --enable-debug<br />
|| meson configure|setup -Ddebug=true<br />
|-<br />
|| specify compiler<br />
|| CC=compiler ./configure<br />
|| CC=compiler meson setup <br />
|| CC is only checked during meson setup, not with meson configure<br />
|-<br />
|| set CFLAGS<br />
|| CFLAGS=options ./configure<br />
|| meson configure|setup -Dc_args=options<br />
|| CFLAGS is also checked, but only for meson setup<br />
|-<br />
|| build<br />
|| make -s<br />
|| ninja<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| build, showing compiler commands<br />
|| make<br />
|| ninja -v<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| install all the binaries and libraries<br />
|| make install<br />
|| ninja install<br />
|| use meson install --quiet for a less verbose experience<br />
|-<br />
|| clean build<br />
|| make clean<br />
|| ninja clean<br />
|| ninja uses parallelism by default, launch from the root of the build tree.<br />
|-<br />
|| run all tests<br />
|| make check-world<br />
|| meson test<br />
|| runs all test, using parallelism by default<br />
|-<br />
|| build documentation<br />
|| cd doc/ && make html && make man<br />
|| ninja docs<br />
|| Builds html documentation and man pages <br />
|-<br />
|| run main regression tests<br />
|| make check<br />
|| meson test --suite setup --suite regress<br />
||<br />
|-<br />
|| run main regression tests against existing server<br />
|| make installcheck<br />
|| meson test --setup running<br />
||<br />
|-<br />
|| list tests<br />
||<br />
|| meson test --list<br />
|}<br />
<br />
=== PostgreSQL devel documentation ===<br />
<br />
[https://www.postgresql.org/docs/devel/install-meson.html "Building and Installation with Meson" section]<br />
<br />
=== Other Notes ===<br />
<br />
ninja tries to run from the root of the build directory. If you are not in the build directory, you can use the "-C" flag to have ninja "change directory" and run from there, e.g.: <br />
<br />
<pre><br />
ninja -C $builddir<br />
</pre><br />
<br />
== Installing Meson ==<br />
<br />
=== Linux ===<br />
<br />
Debian / Ubuntu:<br />
<pre><br />
apt-get update && apt-get install -y meson ninja-build<br />
</pre><br />
<br />
Fedora:<br />
<pre><br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 8:<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled powertools<br />
dnf -y install meson ninja-build<br />
</pre><br />
<br />
RHEL 9 (tested on Rocky Linux 9):<br />
<pre><br />
dnf -y install dnf-plugins-core<br />
dnf config-manager --set-enabled crb<br />
dnf -y install meson<br />
</pre><br />
<br />
=== macOS ===<br />
<br />
With MacPorts:<br />
<br />
<pre><br />
sudo port install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find MacPorts libraries:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br />
</pre><br />
<br />
With Homebrew:<br />
<pre><br />
brew install meson<br />
</pre><br />
<br />
Arguments to meson setup/configure to find Homebrew libraries:<br />
<br />
On arm64:<br />
<pre><br />
meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br />
</pre><br />
<br />
On x86-64:<br />
<pre><br />
meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br />
</pre><br />
<br />
=== Windows ===<br />
<br />
Assuming python is installed, the easiest way to get meson and ninja is:<br />
<pre><br />
pip install meson ninja<br />
</pre><br />
<br />
== Why and What ==<br />
<br />
Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br />
it. Recursive make has a lot of hard to resolve dependency issues and slow<br />
incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br />
developers not using windows and runs tests serially. While these and other<br />
issues could individually be addressed with incremental improvements, together<br />
they seem best addressed by moving to a more modern buildsystem.<br />
<br />
After evaluating different buildsystem choices, we chose to use meson, to a<br />
good degree based on the adoption by other open source projects.<br />
<br />
We decided that it's more realistic to commit a relatively early version of<br />
the new buildsystem and mature it in tree.<br />
<br />
The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br />
after reaching feature parity. However, we're not planning to remove the<br />
autoconf/make buildsystem in the near future. Likely we're going to keep at<br />
least the parts required for PGXS to keep working around until all supported<br />
versions build with meson.<br />
<br />
<br />
== Meson documentation ==<br />
<br />
* [https://mesonbuild.com/Commands.html meson commandline commands]<br />
* [https://mesonbuild.com/Syntax.html meson syntax]<br />
* [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br />
<br />
<br />
== Development tree, other resources ==<br />
<br />
* https://github.com/anarazel/postgres/tree/meson<br />
* https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br />
<br />
== Visualizing builds ==<br />
<br />
When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37417Freezing/skipping strategies patch: motivating examples2022-12-18T23:59:58Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related work added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
Note that the patch <b>completely removes</b> aggressive mode VACUUM. Antiwraparound autovacuums will still exist, but become much rarer. Antiwraparound autovacuums should only be needed in true emergencies with this work in place. <br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<strike><b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b></strike>).<br />
<br />
<b>Update:</b> v10 of the patch series [https://www.postgresql.org/message-id/CAH2-WzmjHQJ7pbdO4BtWVJ6CLG-Mp9CNe914WUJdiScOTNRKRw@mail.gmail.com avoids the freeze spike] that you see here (here we show v9 behavior). So in v10 the same-catch up process will only happen during some much later insert-driven autovacuum, when the pgbench_history table has become far larger (while eager freezing would kick in at the same point as in v9). The <i>relative</i> cost of catch-up freezing won't be too great under this improved scheme; we expect to have no more than about an extra 5% of rel_pages to freeze when it happens now (far less than the 52% of rel_pages shown here on a percentage basis, though not in absolute terms). <br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While relfrozenxid advanced significantly less than it did in the earlier pgbench_history example, it nevertheless advanced by a huge amount by any traditional measure (in particular, by much more than the vacuum_freeze_min_age-based cutoff requires).<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has only a loose relationship with freeze debt</b>, which is hard to predict but tends to be fairly fixed for a given table/workload. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37416Freezing/skipping strategies patch: motivating examples2022-12-18T23:54:22Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related work added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
Note that the patch <b>completely removes</b> aggressive mode VACUUM. Antiwraparound autovacuums will still exist, but become much rarer. Antiwraparound autovacuums should only be needed in true emergencies with this work in place. <br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<strike><b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b></strike>).<br />
<br />
<b>Update:</b> v10 of the patch series [https://www.postgresql.org/message-id/CAH2-WzmjHQJ7pbdO4BtWVJ6CLG-Mp9CNe914WUJdiScOTNRKRw@mail.gmail.com avoids the freeze spike] that you see here (here we show v9 behavior). So in v10 the same-catch up process will only happen during some much later insert-driven autovacuum, when the pgbench_history table has become far larger (while eager freezing would kick in at the same point as in v9). The <i>relative</i> cost of catch-up freezing won't be too great under this improved scheme; we expect to have no more than about an extra 5% of rel_pages to freeze when it happens now (far less than the 50% of rel_pages shown here on a percentage basis, though not in absolute terms). <br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While relfrozenxid advanced significantly less than it did in the earlier pgbench_history example, it nevertheless advanced by a huge amount by any traditional measure (in particular, by much more than the vacuum_freeze_min_age-based cutoff requires).<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has only a loose relationship with freeze debt</b>, which is hard to predict but tends to be fairly fixed for a given table/workload. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37415Freezing/skipping strategies patch: motivating examples2022-12-18T23:53:29Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related work added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
Note that the patch <b>completely removes</b> aggressive mode VACUUM. Antiwraparound autovacuums will still exist, but become much rarer. Antiwraparound autovacuums should only be needed in true emergencies with this work in place. <br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
<b>Update:</b> v10 of the patch series [https://www.postgresql.org/message-id/CAH2-WzmjHQJ7pbdO4BtWVJ6CLG-Mp9CNe914WUJdiScOTNRKRw@mail.gmail.com avoids the freeze spike] that you see here (here we show v9 behavior). So in v10 the same-catch up process will only happen during some much later insert-driven autovacuum, when the pgbench_history table has become far larger (while eager freezing would kick in at the same point as in v9). The <i>relative</i> cost of catch-up freezing won't be too great under this improved scheme; we expect to have no more than about an extra 5% of rel_pages to freeze when it happens now (far less than the 50% of rel_pages shown here on a percentage basis, though not in absolute terms). <br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While relfrozenxid advanced significantly less than it did in the earlier pgbench_history example, it nevertheless advanced by a huge amount by any traditional measure (in particular, by much more than the vacuum_freeze_min_age-based cutoff requires).<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has only a loose relationship with freeze debt</b>, which is hard to predict but tends to be fairly fixed for a given table/workload. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37410Freezing/skipping strategies patch: motivating examples2022-12-15T20:33:36Z<p>Pgeoghegan: </p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related work added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
Note that the patch <b>completely removes</b> aggressive mode VACUUM. Antiwraparound autovacuums will still exist, but become much rarer. Antiwraparound autovacuums should only be needed in true emergencies with this work in place. <br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While relfrozenxid advanced significantly less than it did in the earlier pgbench_history example, it nevertheless advanced by a huge amount by any traditional measure (in particular, by much more than the vacuum_freeze_min_age-based cutoff requires).<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has only a loose relationship with freeze debt</b>, which is hard to predict but tends to be fairly fixed for a given table/workload. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37409Freezing/skipping strategies patch: motivating examples2022-12-15T17:53:32Z<p>Pgeoghegan: </p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
Note that the patch <b>completely removes</b> aggressive mode VACUUM. Antiwraparound autovacuums will still exist, but become much rarer. Antiwraparound autovacuums should only be needed in true emergencies with this work in place. <br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While relfrozenxid advanced significantly less than it did in the earlier pgbench_history example, it nevertheless advanced by a huge amount by any traditional measure (in particular, by much more than the vacuum_freeze_min_age-based cutoff requires).<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has only a loose relationship with freeze debt</b>, which is hard to predict but tends to be fairly fixed for a given table/workload. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37408Freezing/skipping strategies patch: motivating examples2022-12-15T17:39:58Z<p>Pgeoghegan: /* Constantly updated tables (usually smaller tables) */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While relfrozenxid advanced significantly less than it did in the earlier pgbench_history example, it nevertheless advanced by a huge amount by any traditional measure (in particular, by much more than the vacuum_freeze_min_age-based cutoff requires).<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has only a loose relationship with freeze debt</b>, which is hard to predict but tends to be fairly fixed for a given table/workload. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37407Freezing/skipping strategies patch: motivating examples2022-12-15T17:30:22Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While relfrozenxid advanced significantly less than it did in the earlier pgbench_history example, it nevertheless advanced by a huge amount by any traditional measure (in particular, by much more than the vacuum_freeze_min_age-based cutoff requires).<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has a very loose relationship to freeze debt itself</b>. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37406Freezing/skipping strategies patch: motivating examples2022-12-15T17:24:40Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While the final relfrozenxid values we set here is significantly older than the removable<br />
cutoff/OldestXmin if we compare this example to the pgbench_history example, the relfrozenxid is nevertheless very close<br />
to removable cutoff/OldestXmin by any traditional measure.<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has the "use oldest extant XID for relfrozenxid" mechanism added by commit {{PgCommitURL|0b018fab}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has a very loose relationship to freeze debt itself</b>. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37405Freezing/skipping strategies patch: motivating examples2022-12-14T22:33:57Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While the final relfrozenxid values we set here is significantly older than the removable<br />
cutoff/OldestXmin if we compare this example to the pgbench_history example, the relfrozenxid is nevertheless very close<br />
to removable cutoff/OldestXmin by any traditional measure.<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 60% - 65% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has related work added by commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has a very loose relationship to freeze debt itself</b>. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37404Freezing/skipping strategies patch: motivating examples2022-12-14T22:13:12Z<p>Pgeoghegan: /* Mixed inserts and deletes */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While the final relfrozenxid values we set here is significantly older than the removable<br />
cutoff/OldestXmin if we compare this example to the pgbench_history example, the relfrozenxid is nevertheless very close<br />
to removable cutoff/OldestXmin by any traditional measure.<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is fixed (for a given scale factor/number of warehouses), a little like a FIFO queue (though not quite).<br />
Autovacuum tends to need to remove quite a lot of concentrated bloat from the new orders table, to deal with its constant deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 55% - 60% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has related work added by commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has a very loose relationship to freeze debt itself</b>. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37403Freezing/skipping strategies patch: motivating examples2022-12-14T22:10:28Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum (shown here) is representative of every other future autovacuum against the same pgbench_history table, at least on a "percentage of pages scanned/frozen from table" basis:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM. This totally obviates the need for a distinct aggressive mode of operation for VACUUM (and so the patch fully removes the concept of aggressive mode VACUUM, while retaining the concept of antiwraparound autovacuum).<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While the final relfrozenxid values we set here is significantly older than the removable<br />
cutoff/OldestXmin if we compare this example to the pgbench_history example, the relfrozenxid is nevertheless very close<br />
to removable cutoff/OldestXmin by any traditional measure.<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is bound (for a given scale factor/number of warehouses), and autovacuum tends to<br />
need to remove quite a lot of concentrated bloat due to deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 55% - 60% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has related work added by commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has a very loose relationship to freeze debt itself</b>. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37402Freezing/skipping strategies patch: motivating examples2022-12-14T17:11:56Z<p>Pgeoghegan: /* Today, on Postgres HEAD */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum is representative of every other autovacuum that will every happen on the same table going forward (on a "percentage of pages scanned/frozen from table" basis):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM.<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples have already attained an age exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
<b>do less freezing as a consequence of doing more vacuuming!</b><br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While the final relfrozenxid values we set here is significantly older than the removable<br />
cutoff/OldestXmin if we compare this example to the pgbench_history example, the relfrozenxid is nevertheless very close<br />
to removable cutoff/OldestXmin by any traditional measure.<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is bound (for a given scale factor/number of warehouses), and autovacuum tends to<br />
need to remove quite a lot of concentrated bloat due to deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 55% - 60% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has related work added by commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has a very loose relationship to freeze debt itself</b>. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeogheganhttps://wiki.postgresql.org/index.php?title=Freezing/skipping_strategies_patch:_motivating_examples&diff=37401Freezing/skipping strategies patch: motivating examples2022-12-14T17:08:58Z<p>Pgeoghegan: /* Patch */</p>
<hr />
<div>This page documents problem cases addressed by the patch to remove aggressive mode VACUUM by making VACUUM freeze on a<br />
proactive timeline, driven by concerns about managing the number of unfrozen heap pages that have accumulated in larger<br />
tables. This patch is proposed for Postgres 16, and builds on related added to Postgres 15 (see Postgres 15 commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
See also: [https://commitfest.postgresql.org/40/3843/ CF entry for patch series]<br />
<br />
It makes sense to discuss the patch series by focusing on various motivating examples, each of which involves one particular table, with its own more or less fixed set of performance characteristics. VACUUM must decide certain details around freezing and advancing relfrozenxid based on these kinds of per-table characteristics. Certain approaches are really only interesting with certain kinds of tables. Naturally this can change over time, for whatever reason. VACUUM is supposed to keep up with and even anticipate the needs of the table, over time and across multiple successive VACUUM operations.<br />
<br />
= Background: How the visibility map influences the interpretation/application of vacuum_freeze_min_age =<br />
<br />
At a high level, VACUUM currently chooses when and how to freeze tuples based solely on whether or not a given XID is<br />
older than vacuum_freeze_min_age for a tuple on a page that it actually scans. This means that all-visible pages left<br />
behind by a previous VACUUM operation won't even be considered for freezing until the next aggressive mode VACUUM<br />
(barring tuples on heap pages that happen to be modified some time before aggressive VACUUM finally kicks in).<br />
<br />
The introduction of the visibility map in Postgres 8.4 made the mechanism that chooses how to freeze stop reliably<br />
freezing XIDs that attain an age that exceeds the vacuum_freeze_min_age settings. Sometimes it actually does work in<br />
the way that the very earliest design for vacuum_freeze_min_age intended, and sometimes it doesn't, mostly due to the<br />
confounding influence of the visibility map.<br />
<br />
The pre-visibility-map design was based on the idea that lazy processing could avoid needlessly freezing tuples that<br />
would inevitably be modified before too long anyway. That in itself wasn't a bad idea, and still isn't now; laziness<br />
can still make sense. It happens to be <b>wildly inappropriate</b> in certain kinds of tables, where VACUUM should prefer a<br />
much more <b>proactive freezing cadence</b>. Knowing the difference (recognizing the kinds of tables that VACUUM should prefer<br />
to freeze eagerly rather than lazily) is an issue of central importance for the project.<br />
<br />
= Examples =<br />
<br />
Most of these examples show cases where VACUUM behaves lazily, when it clearly should behave eagerly. Even an<br />
expert DBA will currently have a hard time tuning the system to do the right thing with tables/workloads like these ones.<br />
<br />
== Simple append-only ==<br />
<br />
The best example is also the simplest: a strict append-only table, such as pgbench_history. Naturally, this table will<br />
get a lot of insert-driven (triggered by autovacuum_vacuum_insert_threshold) autovacuums.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Currently, we see autovacuum behavior like this (triggered by autovacuum_vacuum_insert_threshold):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 171638 remain, 21942 scanned (12.78% of total)<br />
tuples: 0 removed, 26947118 remain, 0 are dead but not yet removable<br />
removable cutoff: 101761411, which was 767958 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 43958 hits, 1 misses, 1 dirtied<br />
WAL usage: 21461 records, 1 full page images, 1274525 bytes<br />
system usage: CPU: user: 1.35 s, system: 0.34 s, elapsed: 2.39 s<br />
<br />
Note that there are no pages frozen. There will <b>never</b> be <i>any</i> pages frozen by any VACUUM, unless and until the VACUUM is<br />
aggressive, at which point we'll rewrite most of the table to freeze most of its tuples. Clearly this doesn't make any<br />
sense; we really should be eagerly freezing the table from a fairly early stage, so that the burden of freezing is<br />
evenly spread out over time and across multiple VACUUM operations.<br />
<br />
Perhaps there is some limited argument to be made for laziness here, at least earlier on, but why should we solely rely<br />
on table age (aggressive mode VACUUM) to take care of freezing? We need physical units to make a sensible choice in<br />
favor of lazy freezing. After all, table age tells us precisely nothing about the eventual cost of freezing. If the<br />
pgbench_history table happened to have tuples that were twice as wide, that would mean that the eventual cost of<br />
freezing during an aggressive mode VACUUM would also approximately double. But (assuming that all other things remain<br />
unchanged) the timeline for when we'd freeze would stay exactly the same.<br />
<br />
By the same token, the timeline for freezing all of the pages from the pgbench_history table will effectively be<br />
accelerated by transactions that don't even modify the table itself. This isn't fundamentally unreasonable in extreme<br />
cases, where the risk of the system entering xidStopLimit mode really does need to influence the timeline for freezing a<br />
table like this. But we don't just allow table age to determine when we freeze all these pgbench_history pages in<br />
extreme cases -- it is the sole factor that determines when it happens, in practice, including when there is almost no<br />
practical risk of the system entering xidStopLimit (i.e. almost always).<br />
<br />
It's possible to tune vacuum_freeze_min_age in order to make sure that such a table is frozen eagerly, as mentioned in passing by the<br />
[https://www.postgresql.org/docs/current/routine-vacuuming.html#AUTOVACUUM Routine Vacuuming/the Autovacuum Daemon] section of the docs (starting at <code>it may be beneficial to lower the table's autovacuum_freeze_min_age...</code>), but this relies on the DBA actually having a direct understanding of<br />
the problem. It also requires that the DBA use a setting based on XID age (vacuum_freeze_min_age, or the autovacuum_freeze_min_age reloption). While that is at<br />
least feasible for a pure append-only table like this one, it still isn't a particularly natural way for the DBA to<br />
control the problem. (More importantly, tuning vacuum_freeze_min_age with this goal in mind runs into problems with<br />
tables that grow and grow, but have a mix of inserts and updates -- see later examples for more.)<br />
<br />
=== Patch ===<br />
<br />
The patch will have autovacuum/VACUUM consistently freeze all of the pages from the table on an eager timeline (autovacuum is once again triggered by autovacuum_vacuum_insert_threshold here, of course). Initially the patch behaves in a way that isn't visibly different to Postgres HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 477149 remain, 48188 scanned (10.10% of total)<br />
tuples: 0 removed, 74912386 remain, 0 are dead but not yet removable<br />
removable cutoff: 149764963, which was 1759214 XIDs old when operation ended<br />
frozen: 0 pages from table (0.00% of total) had 0 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.004 ms, write: 0.000 ms<br />
avg read rate: 0.001 MB/s, avg write rate: 0.001 MB/s<br />
buffer usage: 96544 hits, 1 misses, 1 dirtied<br />
WAL usage: 47945 records, 1 full page images, 2837081 bytes<br />
system usage: CPU: user: 3.00 s, system: 0.91 s, elapsed: 5.47 s<br />
<br />
The next autovacuum that takes place happens to be the first autovacuum after pgbench_history has<br />
crossed 4GB in size. This threshold is controlled by the GUC vacuum_freeze_strategy_threshold, and its proposed default<br />
is 4GB.<br />
<br />
Here is where the patch starts to diverge from HEAD:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 526836 remain, 49931 scanned (9.48% of total)<br />
tuples: 0 removed, 82713253 remain, 0 are dead but not yet removable<br />
removable cutoff: 157563960, which was 1846970 XIDs old when operation ended<br />
frozen: <b>49665 pages</b> from table (9.43% of total) had 7797405 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.000 ms<br />
avg read rate: 0.003 MB/s, avg write rate: 0.003 MB/s<br />
buffer usage: 100044 hits, 2 misses, 2 dirtied<br />
WAL usage: 99331 records, 2 full page images, 21720187 bytes<br />
system usage: CPU: user: 3.24 s, system: 0.87 s, elapsed: 5.72 s<br />
<br />
Note that this isn't such a huge shift -- not at first. We do freeze all of the pages we've scanned in this VACUUM, but<br />
we don't advance relfrozenxid proactively. A transition is now underway, which finishes when the size of the<br />
pgbench_history table reaches 8GB (since that's twice the vacuum_freeze_strategy_threshold setting). Several more<br />
autovacuums will be triggered that each look similar to the above example.<br />
<br />
Our "transition from lazy to eager strategies" concludes with an autovacuum that actually advanced relfrozenxid eagerly:<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1078444 remain, <b>561143 scanned</b> (52.03% of total)<br />
tuples: 0 removed, 169315499 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>244160328</b>, which was 32167955 XIDs old when operation ended<br />
new relfrozenxid: <b>99467843</b>, which is 24578353 XIDs ahead of previous value<br />
frozen: <b>560841 pages</b> from table (52.00% of total) had 88051825 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 384.224 ms, write: 1003.192 ms<br />
avg read rate: 16.728 MB/s, avg write rate: 36.910 MB/s<br />
buffer usage: 906525 hits, 216130 misses, 476907 dirtied<br />
WAL usage: 1121683 records, 557662 full page images, 4632208091 bytes<br />
system usage: CPU: user: 23.78 s, system: 8.52 s, elapsed: 100.94 s<br />
<br />
This is a little like an aggressive VACUUM, because we have to freeze about half the pages in the table (<b>Note to self: might be a<br />
good idea for the patch to adjust the heuristic so that we advance relfrozenxid eagerly for the first time in a much<br />
later VACUUM -- even this seems a bit too close to aggressive VACUUM</b>).<br />
<br />
From this point onwards every autovacuum of pgbench_history will scan the same percentage of the table's pages, and freeze almost all of those same pages (setting them all-frozen for good). The overhead of the very next autovacuum is representative of every other autovacuum that will every happen on the same table going forward (on a "percentage of pages scanned/frozen from table" basis):<br />
<br />
automatic vacuum of table "regression.public.pgbench_history": index scans: 0<br />
pages: 0 removed, 1500396 remain, 146832 scanned (9.79% of total)<br />
tuples: 0 removed, 235561936 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>310431281</b>, which was 5275067 XIDs old when operation ended<br />
new relfrozenxid: <b>310426654</b>, which is 23032061 XIDs ahead of previous value<br />
frozen: 146698 pages from table (9.78% of total) had 23031179 tuples frozen<br />
index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed<br />
I/O timings: read: 0.013 ms, write: 0.009 ms<br />
avg read rate: 0.002 MB/s, avg write rate: 0.002 MB/s<br />
buffer usage: 294142 hits, 4 misses, 4 dirtied<br />
WAL usage: 293397 records, 4 full page images, 64139188 bytes<br />
system usage: CPU: user: 9.42 s, system: 2.49 s, elapsed: 16.46 s<br />
<br />
In summary, we get <b>perfect performance stability</b> with the patch, after an initial period of adjusting to using an eager approach to freezing in each VACUUM.<br />
<br />
This last autovacuum doesn't have <i>exactly</i> the same details as an autovacuum that simply had vacuum_freeze_min_age set to 0. Note that<br />
there are a tiny number of unfrozen heap pages that still got scanned (146832 scanned - 146698 frozen = <b>134 pages scanned but left unfrozen</b>). We'd probably see no remaining unfrozen scanned pages whatsoever, were we to try the same thing with vacuum_freeze_min_age/autovacuum_freeze_min_age set to 0 -- so what we see here isn't "maximally aggressive" freezing. (Note also that "new relfrozenxid" is not quite the same as "removable cutoff", for the same exact reason -- "maximally aggressive" vacuuming would have been able to get it right up to the "removable cutoff" value shown.)<br />
<br />
VACUUM leaves behind a tiny number of unfrozen pages like this because the patch only triggers page-level freezing<br />
proactively when it sees that the whole heap page will thereby become all-frozen instead of all-visible. So eager<br />
freezing is only a policy about when and how we freeze pages -- it still requires individual heap pages to look a certain way before we'll actually go ahead with freezing them. This aspect of the design barely matters in this example, but will be much more important with the next example.<br />
<br />
== Continual inserts with updates ==<br />
<br />
The TPC-C benchmark has two tables that continue to grow for as long as the benchmark is run: the order table, and the<br />
order lines table. The order lines table is the bigger of the two, by far (each order has about 10 order lines on<br />
average, plus the rows are naturally wider). And these tables are by far the largest tables out of the whole set (at<br />
least after the benchmark is run for a little while, which is a requirement).<br />
<br />
The benchmark is designed to work in a way that is at least loosely based on real world conditions for a network of<br />
wholesalers/distributors. Orders come in from customers, and are delivered some time later; more than 10 hours later<br />
with spec-compliant settings. It's somewhat synthetic data (due to the requirement that the benchmark can be easily scaled up and down), but its design is nevertheless somewhat grounded in physical reality. There can only be so many orders per hour per warehouse. An individual customer can only order so many things per day, because individual human beings can only engage in so many transactions in a given 24 hour period. In short, the benchmark shouldn't have a workload that is wildly unrealistic for the business process that it seeks to simulate.<br />
<br />
See also: [https://github.com/wieck/benchmarksql/blob/master/docs/TimedDriver.md BenchmarkSQL Timed Driver docs], written by Jan Wieck.<br />
<br />
The orders are initially inserted by the order transaction, which will insert rows into both tables in the obvious way.<br />
Later on, all of the rows inserted (into both tables) are updated by the delivery transaction. After that, the<br />
benchmark will never update or delete the same rows, ever again. This could be described as an adversarial workload,<br />
because there is a relatively high number of updates around the same key space/physical heap blocks at the same time,<br />
but the hot-spot continually changes as time marches forward. This adversarial mix is particularly relevant to the<br />
project, because there are two opposite trends that pull in opposite directions -- we want to freeze eagerly in some<br />
pages, but we also want to freeze lazily in other pages. It's relatively difficult for the patch to infer which<br />
approach will work best at the level of each heap page.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
Like the pgbench_history example, we see practically no freezing in non-aggressive VACUUMs for this, before the point<br />
that an aggressive mode VACUUM is required (there are quite a few autovacuums that look like similar to this one from<br />
earlier):<br />
<br />
ts: 2022-12-06 03:18:18 PST x: 0 v: 4/8515 p: 554727 LOG: <b>automatic vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 16784562 remain, <b>4547881 scanned (27.10% of total)</b><br />
tuples: 10112936 removed, 1023712482 remain, 5311068 are dead but not yet removable<br />
removable cutoff: 194441762, which was 7896967 XIDs old when operation ended<br />
frozen: 152106 pages from table <b>(0.91% of total)</b> had 3757982 tuples frozen<br />
index scan needed: 547657 pages from table (3.26% of total) had 1828207 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4448447 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 471323.673 ms, write: 23125.683 ms<br />
avg read rate: 35.226 MB/s, avg write rate: 14.049 MB/s<br />
buffer usage: 4666574 hits, 9431082 misses, 3761396 dirtied<br />
WAL usage: 4587410 records, 2030544 full page images, 13789178294 bytes<br />
system usage: CPU: user: 56.99 s, system: 74.71 s, elapsed: 2091.63 s<br />
<br />
The burden of freezing is almost completely borne by aggressive mode VACUUMs:<br />
<br />
ts: 2022-12-06 06:09:06 PST x: 0 v: 45/8157 p: 556501 LOG: <b>automatic aggressive vacuum</b> of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 18298517 remain, <b>16577256 scanned (90.59% of total)</b><br />
tuples: 12190981 removed, 1127721257 remain, 17548258 are dead but not yet removable<br />
removable cutoff: 213459729, which was 25528936 XIDs old when operation ended<br />
new relfrozenxid: 163469053, which is 148467571 XIDs ahead of previous value<br />
frozen: 10940612 pages from table <b>(59.79% of total)</b> had 640281019 tuples frozen<br />
index scan needed: 420621 pages from table (2.30% of total) had 1345098 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 5219724 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 558603.794 ms, write: 61424.318 ms<br />
avg read rate: 23.087 MB/s, avg write rate: 16.189 MB/s<br />
buffer usage: 16666051 hits, 22135595 misses, 15521445 dirtied<br />
WAL usage: 25282446 records, 12943048 full page images, 89680003763 bytes<br />
system usage: CPU: user: 216.91 s, system: 298.49 s, elapsed: 7490.56 s<br />
<br />
Workload characteristics make it particularly hard to tune VACUUM for such a table. By setting vacuum_freeze_min_age to<br />
0, we'll freeze a lot of tuples that are bound to be updated before long anyway. <br />
<br />
It's possible that we'd see more freezing by vacuuming less often (which is possible by lowering<br />
autovacuum_vacuum_insert_threshold), since that would mean that we'd tend to only reach new heap pages some time after<br />
their tuples has already attained an age already exceeding vacuum_freeze_min_age. This effect is just perverse; we'll<br />
do less freezing as a consequence of doing more vacuuming!<br />
<br />
=== Patch ===<br />
<br />
As with the earlier example, the patch will have autovacuum/VACUUM consistently freeze all of the pages from the table containing<br />
only all-visible tuples right away, so that they're marked all-frozen in the VM instead of all-visible.<br />
<br />
Here we show an autovacuum of the same table, at approximately the same time into the benchmark as the example for<br />
Postgres HEAD (note that the "removable cutoff" XID is close-ish, and that the table is around the same size as it was when we looked at the Postgres HEAD aggressive mode VACUUM):<br />
<br />
ts: 2022-12-05 20:05:18 PST x: 0 v: 43/13665 p: 544950 LOG: automatic vacuum of table "regression.public.bmsql_order_line": index scans: 1<br />
pages: 0 removed, 17894854 remain, <b>2981204 scanned (16.66% of total)</b><br />
tuples: 10365170 removed, 1088378159 remain, 3299160 are dead but not yet removable<br />
removable cutoff: 208354487, which was 7638618 XIDs old when operation ended<br />
new relfrozenxid: 183132069, which is 15812161 XIDs ahead of previous value<br />
frozen: 2355230 pages from table <b>(13.16% of total)</b> had 138233294 tuples frozen<br />
index scan needed: 571461 pages from table (3.19% of total) had 1927325 dead item identifiers removed<br />
index "bmsql_order_line_pkey": pages: 4730173 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 431408.682 ms, write: 29564.972 ms<br />
avg read rate: 30.057 MB/s, avg write rate: 12.806 MB/s<br />
buffer usage: 3048463 hits, 8221521 misses, 3502946 dirtied<br />
WAL usage: 6888355 records, 3056776 full page images, 20240523796 bytes<br />
system usage: CPU: user: 71.84 s, system: 92.74 s, elapsed: 2136.97 s<br />
<br />
Note how we freeze most pages, but still leave a significant number unfrozen each time, despite using an eager approach<br />
to freezing (2981204 scanned - 2355230 frozen = <b>625974 pages scanned but left unfrozen</b>). Again, this is because we<br />
don't freeze pages unless they're already eligible to be set all-visible. We saw the same effect with the first<br />
pgbench_history example, but it was hardly noticeable at all there. Whereas here we see that even eager freezing opts<br />
to hold off on freezing relatively many individual heap pages, due to the observed conditions on those particular heap<br />
pages.<br />
<br />
We're likely to be freezing XIDs in an order that only approximately matches XID age order. Despite all this, we still<br />
consistently see final relfrozenxid values that are comfortably within vacuum_freeze_min_age XIDs of the VACUUM's<br />
OldestXmin/removable cutoff. So in practice <b>XID age has absolutely minimal impact</b> on how or when we freeze, in this<br />
example table/workload. While the final relfrozenxid values we set here is significantly older than the removable<br />
cutoff/OldestXmin if we compare this example to the pgbench_history example, the relfrozenxid is nevertheless very close<br />
to removable cutoff/OldestXmin by any traditional measure.<br />
<br />
All earlier autovacuum operations look similar to this one (and all later autovacuums will also look similar). In fact, even <i>much</i> earlier autovacuums that took place when<br />
the table was much smaller show approximately the same <b>percentage</b> of pages scanned and frozen. So even as the table<br />
continues to grow and grow, the details over time remain approximately the same in that sense. Most importantly of all,<br />
there is never any need for an aggressive mode vacuum that does practically all freezing.<br />
<br />
This isn't perfect; some of the work of freezing still goes to waste, despite efforts to avoid it. This can be seen as<br />
the cost of performance stability. We at least avoid the worst impact of it, by conditioning triggering freezing on<br />
all-visible-ness, and by avoiding eager freezing altogether in smaller tables.<br />
<br />
==== Scanned pages, visibility map snapshot ====<br />
<br />
Independent of the issue of freezing and freeze debt, this example also shows how VACUUM <b>tends to scan significantly fewer pages</b> with the patch, compared to Postgres HEAD/master. This is due to the patch replacing vacuumlazy.c's SKIP_PAGES_THRESHOLD mechanism with visibility map snapshots. VACUUM thereby avoids scanning pages that it doesn't need to scan from the start, and also avoids scanning pages whose VM bit was concurrently unset. Unset visibility map bits are potentially an important factor with long running VACUUM operations, such as these. VACUUM is more insulated from the fact that the table continues to change while VACUUM runs, since we "lock in" the pages VACUUM must scan, at the start of each VACUUM.<br />
<br />
In fact, the patch makes the <b>percentage</b> of scanned pages shown each time (for this workload) both lower and very consistent over time, across successive VACUUM operations, even as the table continues to grow indefinitely (at least for this workload, likely for many others besides). This is another example of how the patch series tends to promote performance stability. VM snapshots make very little difference in small tables, but can help quite a lot in large tables.<br />
<br />
Here we show details of all nearby VACUUM operations against the same table, for the same run (these are over an hour apart):<br />
<br />
pages: 0 removed, 13210198 remain, 2031762 scanned (15.38% of total)<br />
pages: 0 removed, 14270140 remain, 2478471 scanned (17.37% of total)<br />
pages: 0 removed, 15359855 remain, 2654325 scanned (17.28% of total)<br />
pages: 0 removed, 16682431 remain, 3022064 scanned (18.12% of total)<br />
pages: 0 removed, 17894854 remain, 2981204 scanned (16.66% of total)<br />
pages: 0 removed, 19442899 remain, 3519116 scanned (18.10% of total)<br />
pages: 0 removed, 20852526 remain, 3452426 scanned (16.56% of total)<br />
<br />
And the same, for Postgres HEAD/master:<br />
<br />
pages: 0 removed, 12563595 remain, 3116086 scanned (24.80% of total)<br />
pages: 0 removed, 13360079 remain, 3329987 scanned (24.92% of total)<br />
pages: 0 removed, 14328923 remain, 3667558 scanned (25.60% of total)<br />
pages: 0 removed, 15567937 remain, 4127052 scanned (26.51% of total)<br />
pages: 0 removed, 16784562 remain, 4547881 scanned (27.10% of total)<br />
pages: 0 removed, 18298517 remain, 16577256 scanned (90.59% of total)<br />
<br />
== Mixed inserts and deletes ==<br />
<br />
Consider a table like TPC-C's new orders table, which is characterized by continual inserts and deletes. The high<br />
watermark number of rows in the table is bound (for a given scale factor/number of warehouses), and autovacuum tends to<br />
need to remove quite a lot of concentrated bloat due to deletes.<br />
<br />
=== Today, on Postgres HEAD ===<br />
<br />
(Not showing Postgres HEAD, since most individual VACUUM operations look just the same as they do with the patch series).<br />
<br />
=== Patch ===<br />
<br />
Most of the time, the behavior with the patch is very similar to Postgres HEAD, since eager freezing is mostly<br />
inappropriate here (in fact we need very little freezing at all, owing to the specifics of the workload). Here is a<br />
typical example:<br />
<br />
ts: 2022-12-05 20:48:45 PST x: 0 v: 43/18087 p: 546852 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83277 remain, 66541 scanned (79.90% of total)<br />
tuples: 2893 removed, 14836119 remain, 2414 are dead but not yet removable<br />
removable cutoff: 226132760, which was 33124 XIDs old when operation ended<br />
frozen: 161 pages from table (0.19% of total) had 27639 tuples frozen<br />
index scan needed: 64030 pages from table (76.89% of total) had 702527 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 65683 in total, 2668 newly deleted, 3565 currently deleted, 2022 reusable<br />
I/O timings: read: 398.127 ms, write: 4.778 ms<br />
avg read rate: 1.089 MB/s, avg write rate: 12.469 MB/s<br />
buffer usage: 289436 hits, 931 misses, 10659 dirtied<br />
WAL usage: 138965 records, 13760 full page images, 86768691 bytes<br />
system usage: CPU: user: 3.59 s, system: 0.02 s, elapsed: 6.67 s<br />
<br />
The system must nevertheless advance relfrozenxid at some point. The patch has heuristics that weigh both costs and<br />
benefits when deciding when to do this. Table age is one consideration -- settings like vacuum_freeze_table_age do<br />
continue to have some influence. But even with a table like this one, that requires very little if any freezing, the<br />
costs also matter.<br />
<br />
Here we see another autovacuum that is triggered to clean up bloat from those deletes (just like every other autovacuum<br />
for this table), but with one or two key differences:<br />
<br />
ts: 2022-12-05 20:54:57 PST x: 0 v: 43/18625 p: 546998 LOG: automatic vacuum of table "regression.public.bmsql_new_order": index scans: 1<br />
pages: 0 removed, 83716 remain, 83680 scanned (99.96% of total)<br />
tuples: 2183 removed, 14989942 remain, 2228 are dead but not yet removable<br />
removable cutoff: <b>227707625</b>, which was 33391 XIDs old when operation ended<br />
new relfrozenxid: <b>190536853</b>, which is 81808797 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 64206 pages from table (76.70% of total) had 676496 dead item identifiers removed<br />
index "bmsql_new_order_pkey": pages: 66537 in total, 2572 newly deleted, 4115 currently deleted, 3516 reusable<br />
I/O timings: read: 693.757 ms, write: 5.070 ms<br />
avg read rate: 21.499 MB/s, avg write rate: 0.058 MB/s<br />
buffer usage: 308003 hits, 18304 misses, 49 dirtied<br />
WAL usage: 137127 records, 2587 full page images, 25969243 bytes<br />
system usage: CPU: user: 3.97 s, system: 0.14 s, elapsed: 6.65 s<br />
<br />
Notice how relfrozenxid advances, and how we scan more pages than last time. This happens during an autovacuum that<br />
takes place at a point where the table's age is about 55% - 60% of the way to the point that autovacuum needs to launch<br />
an antiwraparound autovacuum.<br />
<br />
Here the patch notices that the added cost of advancing relfrozenxid is moderate, though not very low. The logic for<br />
choosing a vmsnap skipping strategy determines that the cost of advancing relfrozenxid now is sufficiently low that it<br />
makes sense to do so. So we advance relfrozenxid here because of a combination of 1.) it being cheap to do so now<br />
(though not exceptionally cheap), and 2.) the fact that table age is starting to become somewhat of a concern (though<br />
certainly not to the extent that VACUUM is forced to advance relfrozenxid).<br />
<br />
Notice that we don't have to freeze any tuples whatsoever here, and yet we still manage to advance relfrozenxid by a<br />
great deal -- it can actually be advanced further than what we'd see in an aggressive VACUUM in Postgres 14 (though<br />
perhaps not Postgres 15, which has related work added by commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}).<br />
<br />
=== Opportunistically advancing relfrozenxid with bursty, real-world workloads ===<br />
<br />
Real world workloads are bursty, whereas benchmarks like TPC-C are designed to produce an unrealistically steady load.<br />
It's likely that there is considerable variation in how each table needs to be vacuumed based on application<br />
characteristics. For example, a once-off bulk deletion is quite possible. Note that the heuristics in play here will<br />
tend to notice when that happens, and will then tend to advance relfrozenxid simply because it happens to be cheap on<br />
that one occasion (though not too cheap), provided table age is already starting to be a concern. In other words VACUUM<br />
has a decent chance of noticing a "naturally occurring" though narrow window of opportunity to advance relfrozenxid inexpensively.<br />
Costs are a big part of the picture here, which has mostly been missing before now.<br />
<br />
== Constantly updated tables (usually smaller tables) ==<br />
<br />
This example shows something that HEAD (and Postgres 15) already get right, following earlier related work (see commits {{PgCommitURL|0b018fab}}, {{PgCommitURL|f3c15cbe}}, and {{PgCommitURL|44fa8488}}) that the new patch series builds on. It's included here because it shows the<br />
continued relevance of lazy strategy freezing. And because it's a good illustration of just how little freezing may be<br />
required to advance relfrozenxid by a great many XIDs, due to workload characteristics naturally present in some types of tables.<br />
<br />
One key observation behind the patch, that recurs again and again, is that <b>relfrozenxid generally has a very loose relationship to freeze debt itself</b>. Understanding and exploiting that difference comes up again and again. It <b>works both ways</b>; sometimes we need to freeze a lot to advance relfrozenxid by a tiny amount, and other times we need to do no freezing whatsoever to advance relfrozenxid by a huge number of XIDs. Here we show an example of the latter case. <br />
<br />
Consider the following totally generic autovacuum output from pgbench's pgbench_branches table (could have used<br />
pgbench_tellers table just as easily):<br />
<br />
automatic vacuum of table "regression.public.pgbench_branches": index scans: 1<br />
pages: 0 removed, 145 remain, 103 scanned (71.03% of total)<br />
tuples: 3464 removed, 129 remain, 0 are dead but not yet removable<br />
removable cutoff: <b>340103448</b>, which was 0 XIDs old when operation ended<br />
new relfrozenxid: <b>340100945</b>, which is 377040 XIDs ahead of previous value<br />
frozen: <b>0 pages</b> from table (0.00% of total) had 0 tuples frozen<br />
index scan needed: 72 pages from table (49.66% of total) had 395 dead item identifiers removed<br />
index "pgbench_branches_pkey": pages: 2 in total, 0 newly deleted, 0 currently deleted, 0 reusable<br />
I/O timings: read: 0.000 ms, write: 0.000 ms<br />
avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s<br />
buffer usage: 304 hits, 0 misses, 0 dirtied<br />
WAL usage: 209 records, 0 full page images, 20627 bytes<br />
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s<br />
<br />
Here we see that autovacuum manages to set relfrozenxid to a very recent XID, despite freezing precisely zero pages.<br />
In practice we'll always see this in any table with similar workload characteristics. Since every tuple is updated before long anyway, no old XID will ever get old enough to need to be frozen, which every VACUUM will notice automatically, which will be reflected in the final relfrozenxid. In practice this seems to happen reliably with tables like this one.<br />
<br />
Although this example involves zero freezing in every VACUUM, and so represents one extreme, there are similar tables/workloads (such as the previous example of the bmsql_new_order table/workload)<br />
that require only a very small amount of freezing to advance relfrozenxid by a great many XIDs -- perhaps just a tiny amount of freezing with negligible cost. To some degree these sorts of scenarios<br />
justify the opportunistic nature of eager strategy vmsnap skipping from the new patch series. VACUUM cannot ever<br />
notice that one particular table has these favorable properties without <i>trying</i> to advance relfrozenxid by some amount,<br />
and then <b>noticing</b> that it can be advanced by a great deal quite easily. (The other reason to be eager about advancing<br />
relfrozenxid is to avoid advancing relfrozenxid for many different tables around the same time.)</div>Pgeoghegan