https://wiki.postgresql.org/api.php?action=feedcontributions&user=Rjuju&feedformat=atomPostgreSQL wiki - User contributions [en]2024-03-28T09:10:44ZUser contributionsMediaWiki 1.35.13https://wiki.postgresql.org/index.php?title=Reviewing_a_Patch&diff=37300Reviewing a Patch2022-10-31T09:59:14Z<p>Rjuju: </p>
<hr />
<div>{{Languages}}<br />
<br />
PostgreSQL development is organized into [[CommitFest]] periods that need plenty of reviewers to help with all the submissions. Many people feel that they're not qualified to do a full review of patches submitted to PostgreSQL. But review includes many different tasks, and even if you can't do all of them you can help the project by taking on the earlier phases. If you can apply a patch and you can use the new feature, you're already qualified to start reviewing it. <br />
<br />
The eventual committer will do their own review before the patch goes into the codebase. The task of a reviewer is to take off load from committers by catching simple problems. The reviewer's job is not necessarily to guarantee some level of quality, but just to report any problems they are able to find. That task is done if you think the patch is ready for in-depth review from a committer. See [http://archives.postgresql.org/pgsql-hackers/2009-07/msg01103.php this patch review] as one example of the output a thorough review might produce. Reviews for other patches might, of course, contain different sections or for that matter, look completely different.<br />
<br />
See also [[Media:11_eggyknap-patch-review.pdf|Josh Tolley's presentation]] (2009) for a tongue-in-cheek but not inaccurate description of who is welcome, and [http://www.pgcon.org/2011/schedule/events/368.en.html Review of Patch Reviewing] (2011) or [https://github.com/dwsteele/conference/blob/release/PostgresPatchReview-PGConfEU-2016/slides/slides.pdf Reviewing PostgreSQL Patches for Fun and Profit] (2016) for more serious looks at the material that's covered here.<br />
<br />
Tactical points:<br />
<br />
* If a commitfest is currently in progress, it is available [https://commitfest.postgresql.org/inprogress here].<br />
* Every commitfest ought to have a commitfest manager that you can ask for help picking a patch or on procedural matters.<br />
** The current commitfest manager is: (vacant)<br />
* You don't have to ask for permission to sign yourself up.<br />
* Please sign up as soon as you know that you plan to do the review. (Some people hold back on signing up until they have actually done the review. Please don't do that; it doesn't help us plan our reviewing resources.)<br />
* On the other hand, we ask that initial reviews are sent in within about five days, to avoid "blocking" reviewing spots. But it's OK to send in a partial review or ask for more time. Just keep communicating.<br />
* Reviews and other development communications should generally be done via the [http://www.postgresql.org/list/pgsql-hackers/ pgsql-hackers] mailing list.<br />
* Send reviews as replies to the email containing the patch. Try to keep the email threading intact. Don't start a new thread for the review, or chances are the original author won't see it.<br />
<br />
Phases a patch review generally goes through include:<br />
<br />
== Submission review (skills needed: patch, English comprehension) ==<br />
<br />
* Is the patch in a patch format which has context? (eg: [http://en.wikipedia.org/wiki/Diff#Context_format context diff format])<br />
** Patches in 'normal' or 'plain' diff formats, which only show the lines changed and no context, are not acceptable.<br />
** Ideally, submitters should choose either context (diff -c) format or unified (diff -u) format based on which makes the submitter's patch easier to read.<br />
* Does it apply cleanly to the current git master?<br />
* Does it include reasonable tests, necessary doc patches, etc?<br />
<br />
== Usability review (skills needed: test-fu, ability to find and read spec) == <br />
<br />
Read what the patch is supposed to do, and consider:<br />
<br />
* Does the patch actually implement that? <br />
* Do we want that? <br />
* Do we already have it? <br />
* Does it follow SQL spec, or the community-agreed behavior? <br />
* Does it include pg_dump support (if applicable)?<br />
* Are there dangers? <br />
* Have all the bases been covered?<br />
<br />
== Feature test (skills needed: patch, configure, make, pipe errors to log) ==<br />
<br />
Apply the patch, compile it and test:<br />
<br />
* Does the feature work as advertised?<br />
* Are there corner cases the author has failed to consider?<br />
* Are there any assertion failures or crashes?<br />
**Reviews should be done with the ''configure'' options<br />
''--enable-cassert'' and ''--enable-debug'' turned on; see [[Working with git]] for a full example. These will help find issues with the code that might otherwise be missed. But note a copy of PostgreSQL built using these parameters will be substantially slower than one built without them. If you're working on something performance-related, such as testing whether a patch slows anything down, be sure to build without these flags before testing execution speed. Some, but not all, of the penalty of ''--enable-cassert'' can be turned off at server start time by putting ''debug_assertions = false'' in your postgresql.conf. See [http://www.postgresql.org/docs/current/static/runtime-config-developer.html Developer Options] for more details about that setting; it defaults to true in builds done with ''--enable-cassert''. Also note that while ''--enable-debug'' shouldn't have any performance penalty when building with ''gcc'', it definitely does with most other compilers.<br />
<br />
== Performance review (skills needed: ability to time performance) ==<br />
<br />
* Does the patch slow down simple tests? <br />
* If it claims to improve performance, does it?<br />
* Does it slow down other things?<br />
<br />
== Coding review (skills needed: guideline comparison, experience with portability issues, minor C-reading skills) ==<br />
<br />
Read the changes to the code in detail and consider:<br />
<br />
* Does it follow the project [http://developer.postgresql.org/pgdocs/postgres/source.html coding guidelines]? <br />
* Are there portability issues? <br />
* Will it work on Windows/BSD etc? <br />
* Are the comments sufficient and accurate?<br />
* Does it do what it says, correctly?<br />
* Does it produce compiler warnings?<br />
* Can you make it crash?<br />
<br />
== Architecture review (skills needed: experience with whole-PostgreSQL-project architecture) ==<br />
<br />
Consider the changes to the code in the context of the project as a whole:<br />
<br />
* Is everything done in a way that fits together coherently with other features/modules? <br />
* Are there interdependencies that can cause problems?<br />
<br />
== Review review ==<br />
<br />
* Did the reviewer cover all the things that kind of reviewer is supposed to do?<br />
<br />
[[Category:Development]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_15_Open_Items&diff=37128PostgreSQL 15 Open Items2022-07-26T05:10:36Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
* [https://www.postgresql.org/message-id/20220603195318.qk4voicqfdhlsnoh@alap3.anarazel.de Reduce amount of logs generated by TAP tests of pg_upgrade?]<br />
** Owner: Michael Paquier<br />
** Other thread: [https://www.postgresql.org/message-id/YrP6ZRXITYWhpVrl@paquier.xyz here]<br />
* [https://www.postgresql.org/message-id/20220616233130.rparivafipt6doj3%40alap3.anarazel.de PG 15 (and to a smaller degree 14) regression due to ExprEvalStep size]<br />
** Owner: Andrew Dunstan<br />
* [https://www.postgresql.org/message-id/20220701231413.GI13040@telsasoft.com large objects lost on upgrade]<br />
** Owner: Robert Haas (9a974cbcba005256a19991203583a94b4f9a21a9)<br />
* [https://www.postgresql.org/message-id/17558-3f6599ffcf52fd4a%40postgresql.org Endless loop with UNIQUE NULLS NOT DISTINCT and INSERT ... ON CONFLICT]<br />
** Owner: Peter Eisentraut (94aa7cc5f707712f592885995a28e018c7c80488)<br />
* [https://www.postgresql.org/message-id/20220726050402.vsr6fmz7rsgpmdz3@jrouhaud wrong filename used in pg_ident_file_mapping infrastructure]<br />
** Owner: Michael Paquier<br />
** Patch available in the report<br />
<br />
<br />
== Decisions to Recheck Mid-Beta ==<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br />
** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer]<br />
** [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report]<br />
** [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report]<br />
<br />
* [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and >MaxAllocSize lengths]<br />
** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br />
** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br />
** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br />
** Affects all stable branches.<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-Wzn22s42h4Lh6v96GsXSKGd%3D_6b76mjqip_WFCGnBmTJCw%40mail.gmail.com CLUSTER sort on abbreviated expressions is broken]<br />
** Affects all stable branches.<br />
** Fixed by: {{PgCommitURL|8ab0ebb9a842dc6063d1374a38b47a3b7ee64afe}}<br />
<br />
* [https://www.postgresql.org/message-id/17485-396609c6925b982d%40postgresql.org Records missing from Primary Key index when doing REINDEX INDEX CONCURRENTLY]<br />
** Affects v14<br />
** Fixed by: {{PgCommitURL|e28bb885196916b0a3d898ae4f2be0e38108d81b}}<br />
<br />
* [https://www.postgresql.org/message-id/20220519193839.GT19626%40telsasoft.com -c min_dynamic_shared_memory now triggers an assertion]<br />
** Affects v14<br />
** Fixed by: {{PgCommitURL|7201cd1862}}<br />
<br />
* [https://www.postgresql.org/message-id/f8a4105f076544c180a87ef0c4822352%40stmuk.bayern.de Extension pg_trgm, permissions and pg_dump order]<br />
** Affects all stable branches.<br />
** Fixed by {{PgCommitURL|00377b9a02b89a831ae50e1c718d34565356698f}}<br />
<br />
== Non-bugs ==<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 15beta3 ===<br />
<br />
* [https://www.postgresql.org/message-id/CAApHDvrHQkiFRHiGiAS-LMOvJN-eK-s762=tVzBz8ZqUea-a_A@mail.gmail.com tuplesort Generation memory contexts don't play nicely with index builds]<br />
** Owner: David Rowley<br />
** Fixed by: {{PgCommitURL|ae1123f9899fe80935ae344e38f18632beb1bf9a}}<br />
* [https://www.postgresql.org/message-id/YrpVkADAY0knF6vM@paquier.xyz Repeatability of installcheck for test_oat_hooks]<br />
** Owner: Andrew Dunstan<br />
** Fixed by: {{PgCommitURL|a6434b951558baad8372dc4b83bf87606dac9cda}}<br />
* [https://www.postgresql.org/message-id/20220530190155.47wr3x2prdwyciah@alap3.anarazel.de Revert debugging added due to 019_replslot_limit]<br />
** Owner: Andres Freund<br />
** Reverted: {{PgCommitURL|3f8148c256e067dc2e8929ed174671ba7dc3339c}}<br />
* [https://www.postgresql.org/message-id/CAApHDvqXpLzav6dUeR5vO_RBh_feHrHMLhigVQXw9jHCyKP9PA%40mail.gmail.com PG15 beta1 sort performance regression due to Generation context change]<br />
** Owner: David Rowley<br />
* [https://www.postgresql.org/message-id/20220706224727.GA2158260@nathanxps13 pg_parameter_aclcheck() and trusted extensions]<br />
** Owner: Tom Lane (a0ffa885e478f5eeacc4e250e35ce25a4740c487)<br />
** Fixed by: {{PgCommitURL|13d83881514856353dc86575eb0fc28132349a60}}<br />
* [https://www.postgresql.org/message-id/YtjsbtZFCaou6C/k@paquier.xyz Unprivileged user can induce crash by using an SUSET param in PGOPTIONS]<br />
** Owner: Tom Lane (a0ffa885e478f5eeacc4e250e35ce25a4740c487)<br />
** Fixed by: {{PgCommitURL|b35617de37870756bdb0e00ffc0a42441e56eefa}}<br />
<br />
=== resolved before 15beta2 ===<br />
<br />
* [https://www.postgresql.org/message-id/CA+HiwqGAGobiiHR8nH382HJxqm1mzZs8=3oKPXnXivWoFSZmNA@mail.gmail.com pgbench --partitions=0]<br />
** Owner; Michael Paquier (6f164e6d17616a157ea5d9e34dbb1b211c080c41)<br />
** Fixed by: {{PgCommitURL|27f1366050c6cd8c1ea5f03b367a5a167ebf34b7}}<br />
* [https://www.postgresql.org/message-id/3813350.1652111765%40sss.pgh.pa.us psql now shows zero elapsed time after an error]<br />
** Owner: Peter Eisentraut<br />
** Fixed by: {{PgCommitURL|9520f8d92a8681e441cc863422babd544353dd39}}<br />
* [https://www.postgresql.org/message-id/17495-7ffe2fa0b261b9fa@postgresql.org Regression in 15beta1 when filtering subquery including row_number window function]<br />
** Owner: David Rowley (9d9c02ccd1aea8e9131d8f4edb21bf1687e40782)<br />
** Fixed by: {{PgCommitURL|3e9abd2eb1b1f6863250f060290f514f30ce8044}}<br />
* [https://www.postgresql.org/message-id/20220524235250.gtt3uu5zktfkr4hv@alap3.anarazel.de Safety of subtrans ID caching]<br />
** Owner: Michael Paquier (06f5295af673df795e8e70e28c43d61c2817b6df)<br />
** Fixed by: {{PgCommitURL|b4529005fd387e863bfa9eb863629b1183c0449c}}<br />
* [https://www.postgresql.org/message-id/f80ace33-11fb-1cd3-20f8-98f51d151088@enterprisedb.com pg_upgrade test writes to source directory]<br />
** Owner: Michael Paquier (322becb6085cb92d3708635eea61b45776bf27b6)<br />
** Fixed by: {{PgCommitURL|15b6d2155375dee2fcba072fffa03c1c8b44656c}}<br />
* [https://www.postgresql.org/message-id/77e6ecaa-2785-97aa-f229-4b6e047cbd2b@enterprisedb.com pg_upgrade is not idempotent, even with --check]<br />
** Owner: Michael Paquier (38bfae36526636ef55daf7cf2a3282403587cb5b)<br />
** Fixed by: {{PgCommitURL|4fff78f00910af0137f9de7532f8eb21d08ab1c3}}<br />
* [https://www.postgresql.org/message-id/202204251548.mudq7jbqnh7r@alvherre.pgsql bogus: logical replication rows/cols combinations]<br />
** Owner: Amit Kapila<br />
** Fixed by: {{PgCommitURL|fd0b9dcebda7b931a41ce5c8e86d13f2efd0af2e}}<br />
* [https://www.postgresql.org/message-id/05ebcb44-f383-86e3-4f31-0a97a55634cf%40enterprisedb.com Ignoring BRIN for HOT udpates seems broken]<br />
** Owner: Tomas Vondra (5753d4ee320b)<br />
** Fixed by: {{PgCommitURL|e3fcca0d0d2414f3a50d6fd40eddf48b7df81475}}<br />
* [https://www.postgresql.org/message-id/PAXPR02MB760039506C87A2083AD85575E3DA9%40PAXPR02MB7600.eurprd02.prod.outlook.com psql no longer reports NOTICE messages promptly]<br />
** Owner: Peter Eisentraut (7844c9918)<br />
** Fixed by: {{PgCommitURL|e77de23fbb0f4ef27090c144edcfa889bb2a06d5}}<br />
* [https://www.postgresql.org/message-id/20220517.162719.1671558681467343711.horikyota.ntt@gmail.com amcheck is using a wrong macro to check compressed-ness]<br />
** Owner: Robert Haas (bd807be6935929bdefe74d1258ca08048f0aafa3)<br />
** Fixed by: {{PgCommitURL|e243de03fb4583dd4a9f0afb41493727d7946c02}}<br />
* [https://www.postgresql.org/message-id/20220607154744.vvmitnqhyxrne5ms%40jrouhaud COPY WITH (HEADER MATCH) broken with custom attribute list]<br />
** Owner: Peter Eisentraut (072132f04e55c1c3b0f1a582318da78de7334379)<br />
** Fixed by: {{PgCommitURL|ca7a0d1d368216e89359c63531a4df0b99a437e4}}<br />
* [https://www.postgresql.org/message-id/flat/DM4PR84MB17349C4E7D88A68264C18AF3EED69%40DM4PR84MB1734.NAMPRD84.PROD.OUTLOOK.COM PG15 beta1 fix pg_stats_ext/pg_stats_ext_exprs view manual]<br />
** Tomas Vondra<br />
** Fixed by: {{PgCommitURL|401f623c7b14890011b9bb9dda7639b1de4d40ad}}<br />
* [https://www.postgresql.org/message-id/20220625151930.GH22452@telsasoft.com Incorrect version check for datlocprovider in pg_upgrade]<br />
** Owner: Peter Eisentraut (f2553d43060edb210b36c63187d52a632448e1d2)<br />
** Fixed by: {{PgCommitURL|fa06a34d14ea053e1e405a6ab2a1c3f1631c3a5e}}<br />
* [https://www.postgresql.org/message-id/17522-bfcd5c603b5f4daa@postgresql.org Failure in TAP tests for IP address support in SANs with LibreSSL]<br />
** Owner: Peter Eisentraut (c1932e542863f0f646f005b3492452acc57c7e66)<br />
** Fixed by: {{PgCommitURL|901a9d53011573e45cd7b87682f0520ef3b0fd2d}}<br />
<br />
=== resolved before 15beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/de57761c-b99b-3435-b0a6-474c72b1149a%40enterprisedb.com libpq: duplicate error message after connection loss]<br />
** Fixed by: {{PgCommitURL|93909599cdba64c8759d646983c0a4ef93de1e50}}<br />
<br />
* [https://www.postgresql.org/message-id/fab3b90a-914d-46a9-beb0-df011ee39ee5%40www.fastmail.com MERGE: ERROR: variable not found in subplan target lists]<br />
** Fixed by: {{PgCommitURL|ce4f46fdc814eb1b704d81640f6d8f03625d0f53}}<br />
<br />
* [https://www.postgresql.org/message-id/20220212211316.GK31460%40telsasoft.com Buildfarm warnings]<br />
** pg_basebackup.c:1261:35: warning: storing the address of local variable archive_filename in progress_filename [-Wdangling-pointer=]<br />
** new in 23a1c6578 - looks like a real error @Robert Haas<br />
** Fixed at: {{PgCommitURL|62cb7427d1e491faf8612a82c2e3711a8cd65422}}<br />
<br />
* [https://www.postgresql.org/message-id/20220311010223.GI28503@telsasoft.com pg_basebackup serverside compression broken with stdout and manifests]<br />
** Fixed at: {{PgCommitURL|b2de45f9200d9adcac50015521574696dc464ccd}}<br />
<br />
* pg_basebackup: bbstreamer_lz4.c:172: bbstreamer_lz4_compressor_content: Assertion `mystreamer->base.bbs_buffer.maxlen >= out_bound' failed. <br />
** [https://www.postgresql.org/message-id/20220316151253.GB28503@telsasoft.com basebackup LZ4 to stdout]<br />
** Owner: Robert Haas (dab298471ff2f91f33bc25bfb73e435d3ab02148)<br />
** Fixed at: {{PgCommitURL|afb529e6772b4e2b065644a2204697eeaf6c9a96}}<br />
<br />
* [https://www.postgresql.org/message-id/CAKFQuwamFuaQHKdhcMt4Gbw5+Hca2UE741B8gOOXoA=TtAd2Yw@mail.gmail.com Incorrect reset timestamp in stats after crash recovery]<br />
** Owner: Andres Freund (5891c7a8ed8f2d3d577e7eea34dacff12d7b6bbd)<br />
** Fixed at: {{PgCommitURL|5cd1c40b3ce9600f129fd1fea9850e1affaf31d5}}<br />
<br />
* [https://www.postgresql.org/message-id/YlPQGNAAa04raObK@paquier.xyz Fixes for compression options of pg_receivewal and refactoring of backup_compression.{c,h}]<br />
** Owner: Michael Paquier (babbbb595d2322da095a1e6703171b3f1f2815cb)<br />
** Fixed at: {{PgCommitURL|042a923ad53dfbe39a9d5012d6c3cf3c9c338884}}<br />
<br />
* [https://www.postgresql.org/message-id/CA+TgmoazKcKUWtqVa0xZqSzbKgTH+X-aw4V7GyLD68EpDLMh8A@mail.gmail.com Remove compatibility from pg_basebackup?]<br />
** Fixed at: {{PgCommitURL|9cd28c2e5f11dfeef64a14035b82e70acead65fd}}<br />
<br />
* [https://www.postgresql.org/message-id/4015413.1649454951%40sss.pgh.pa.us Timing-dependent failure in 002_archiving.pl]<br />
** Owner: Michael Paquier (46dea2419ee7895a4eb3d048317682e6f18a17e1)<br />
** Fixed at: {{PgCommitURL|e61efafcb82c605dcc78f668685223e20d2f7ad8}}, {{PgCommitURL|1a8b110539efe18803c1fa8aa452a2178dbad9a9}}<br />
<br />
* [https://www.postgresql.org/message-id/CA+hUKGJRbzaAOUtBUcjF5hLtaSHnJUqXmtiaLEoi53zeWSizeA@mail.gmail.com qsort performance regression]<br />
** Owner: John Naylor (6974924347c908335607a4a2f252213d58e21b7c)<br />
** Fixed at: {{PgCommitURL|99c754129d787ea4ce3b34b9f4c5f5e74c45ab6a}}<br />
<br />
* [https://www.postgresql.org/message-id/YlZyp26LVVfmwfgW@paquier.xyz Small issues with CLUSTER on partitioned tables]<br />
** Owner: Alvaro Herrera (cfdd03f45e6afc632fbe70519250ec19167d6765)<br />
** Fixed at: {{PgCommitURL|3f19e176ae0f55a653d62e1504dbe5ad8c1006a0}}, {{PgCommitURL|21a10368eb3fce73f146d7e48b4d81496f60d965}}<br />
<br />
* [https://www.postgresql.org/message-id/20220408124338.GK24419@telsasoft.com asynchronous execution crash in trivial_subqueryscan()]<br />
** Owner: Etsuro Fujita (c2bb02bc2e858ba345b8b33f1f3a54628f719d93)<br />
** Fixed at: {{PgCommitURL|5c854e7a2c8a6cd26040e0f9949e7a4a007f6366}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/20220209220004.kb3dgtn2x2k2gtdm%40alap3.anarazel.de Corruption due to relfilenode reuse]<br />
** pg_upgrade can corrupt data with the new OIDs preservation feature<br />
*** Fixed at: {{PgCommitURL|e2f65f42555ff531c6d7c8f151526b4ef7c016f8}}<br />
** the ProcSignalBarrier solution this builds on also turns out to have a small race/hole<br />
*** Fixed at: {{PgCommitURL|b74e94dc27fdbb13954f230b1d1298430afa6c0c}}<br />
** Owner: Thomas Munro, Robert Haas<br />
<br />
* [https://www.postgresql.org/message-id/20220502042718.GB1565149@rfd.leadboat.com Some issues with the TAP tests of pg_upgrade]<br />
** Owner: Michael Paquier<br />
** Fixed at: {{PgCommitURL|7dd3ee508432730d15c5d3032f37362f6b6e4dd8}}<br />
<br />
* [https://www.postgresql.org/message-id/CAMbWs4-LN%3DbF8f9eU2R94dJtF54DfDvBq%2BovqHnOQqbinYDrUw%40mail.gmail.com Crash in _outPathTarget]<br />
** Owner: Peter Eisentraut<br />
** Fixed at: {{PgCommitURL|9ddf251f94090cebf1bd8fc18396cb8a4b580d04}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/Ymd/e5eeZMNAkrXo%40paquier.xyz#23885a148c6899cc874a7bf68f228777 Instability of regression test of pg_walinspect]<br />
** Owner: Jeff Davis<br />
** Fixed at: {{PgCommitURL|ed57cac84d1c5642737dab1e4c4b8cb4f0c4305f}}<br />
<br />
* [https://www.postgresql.org/message-id/YkfeMNYRCGhySKyg%40ahch-to crash with JSON constructors and window functions]<br />
** Owner: Andrew Dunstan (f4fb45d15c59d7add2e1b81a9d477d0119a9691a)<br />
** Fixed at: {{PgCommitURL|4eb9798879680dcc0e3ebb301cf6f925dfa69422}}, {{PgCommitURL|112fdb3528465cc14a2f1dff3dc27f100326d885}}<br />
<br />
* [https://www.postgresql.org/message-id/CAA4eK1LpBFU49Ohbnk%3Ddv_v9YP%2BKqh1%2BSf8i%2B%2B_s-QhD1Gy4Qw%40mail.gmail.com 013_partition.pl failing]<br />
** Fixed at: {{PgCommitURL|dd4ab6fd6528e160571986fa8817cee9f2645aa8}}<br />
<br />
* [https://www.postgresql.org/message-id/Yni6ZHkGotUU+RSf@paquier.xyz Avoid garbage logs with postgres -C on runtime-computed GUCs]<br />
** Fixed at: {{PgCommitURL|8bbf8461a3a2a38ce5f2952a025385b6938a61f7}}<br />
** Owner: Michael Paquier<br />
<br />
* [https://www.postgresql.org/message-id/20220506234924.6mxxotl3xl63db3l@alap3.anarazel.de Some issues with mark_pgdllimport.pl]<br />
** Fixed at: {{PgCommitURL|5edeb574285ecbcc47f0b769a7e363404db0155b}}<br />
** Owner: Robert Haas<br />
<br />
* [https://www.postgresql.org/message-id/1656446.1650043715%40sss.pgh.pa.us Crash in new pgstats code]<br />
** Initially reported issue was fixed by {{PgCommitURL|4a736a161c306fcfed970e6b649f2f03f465ac24}}, but there may be more to do here.<br />
** Owner: Andres Freund<br />
<br />
* [https://www.postgresql.org/message-id/b3463b8c-2328-dcac-0136-af95715493c1%40xs4all.nl TRAP: FailedAssertion("tabstat->trans == trans", File: "pgstat_relation.c", Line: 508]<br />
** Fixed at: {{PgCommitURL|0cf16cb8ca4853b084c40eca310c4c9c3ebf7e2a}}<br />
** Owner: Andres Freund<br />
<br />
* [https://www.postgresql.org/message-id/YlGJGiofZiWN3elx@jrouhaud limitations of GetMaxBackends()]<br />
** Fixed at: {{PgCommitURL|4f2400cb3f10aa79f99fba680c198237da28dd38}}, {{PgCommitURL|ab02d702ef08343fba30d90fdf7df5950063e8c9}}, {{PgCommitURL|7fc0e7de9fb8306e84d1c15211aba4308f694455}}<br />
** Owner: Robert Haas (aa64f23b02924724eafbd9eadbf26d85df30a12b, and 4567596316d186c6e61c72df013797266fcac2f7)<br />
<br />
== Won't Fix ==<br />
<br />
* InvokeNamespaceSearchHook calls need to be moved<br />
** [https://www.postgresql.org/message-id/2600348.1647987525%40sss.pgh.pa.us Re: New Object Access Type hooks]<br />
** Problem showed by 90efa2f5565d28054c30c18f6a2f17f94fdff91e.<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Feature Freeze: April 7, 2022 ('''Last Day to Commit Features''')<br />
* Beta 1: May 19, 2022<br />
* Beta 2: June 30, 2022<br />
* Beta 3: (August 11, 2022)<br />
* GA: TBD<br />
<br />
== See also ==<br />
<br />
* [[Release Management Team]]<br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_15_Open_Items&diff=37041PostgreSQL 15 Open Items2022-06-07T15:50:29Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
* InvokeNamespaceSearchHook calls need to be moved<br />
** [https://www.postgresql.org/message-id/2600348.1647987525%40sss.pgh.pa.us Re: New Object Access Type hooks]<br />
** Owner: Andrew Dunstan (90efa2f5565d28054c30c18f6a2f17f94fdff91e)<br />
* [https://www.postgresql.org/message-id/202204251548.mudq7jbqnh7r@alvherre.pgsql bogus: logical replication rows/cols combinations]<br />
** Owner: Amit Kapila<br />
* [https://www.postgresql.org/message-id/20220517.162719.1671558681467343711.horikyota.ntt@gmail.com amcheck is using a wrong macro to check compressed-ness]<br />
** Owner: Robert Haas (bd807be6935929bdefe74d1258ca08048f0aafa3)<br />
* [https://www.postgresql.org/message-id/CAApHDvqXpLzav6dUeR5vO_RBh_feHrHMLhigVQXw9jHCyKP9PA%40mail.gmail.com PG15 beta1 sort performance regression due to Generation context change]<br />
** Owner: David Rowley<br />
* [https://www.postgresql.org/message-id/20220519193839.GT19626%40telsasoft.com -c min_dynamic_shared_memory now triggers an assertion]<br />
** Owner: Thomas Munro ?<br />
* [https://www.postgresql.org/message-id/05ebcb44-f383-86e3-4f31-0a97a55634cf%40enterprisedb.com Ignoring BRIN for HOT udpates seems broken]<br />
** Owner: Tomas Vondra (5753d4ee320b)<br />
* [https://www.postgresql.org/message-id/PAXPR02MB760039506C87A2083AD85575E3DA9%40PAXPR02MB7600.eurprd02.prod.outlook.com psql no longer reports NOTICE messages promptly]<br />
** Owner: Peter Eisentraut (7844c9918)<br />
* [https://www.postgresql.org/message-id/20220530190155.47wr3x2prdwyciah@alap3.anarazel.de Revert debugging added due to 019_replslot_limit]<br />
** Owner: Andres Freund<br />
* [https://www.postgresql.org/message-id/77e6ecaa-2785-97aa-f229-4b6e047cbd2b@enterprisedb.com pg_upgrade is not idempotent, even with --check]<br />
** Owner: Michael Paquier (38bfae36526636ef55daf7cf2a3282403587cb5b)<br />
* [https://www.postgresql.org/message-id/20220607154744.vvmitnqhyxrne5ms%40jrouhaud COPY WITH (HEADER MATCH) broken with custom attribute list]<br />
** Owner: Peter Eisentraut (072132f04e55c1c3b0f1a582318da78de7334379)<br />
<br />
== Decisions to Recheck Mid-Beta ==<br />
<br />
* Defaults:<br />
** [https://www.postgresql.org/message-id/flat/CALj2ACX-rW_OeDcp4gqrFUAkf1f50Fnh138dmkd0JkvCNQRKGA%40mail.gmail.com log_checkpoints, log_autovacuum_min_duration]<br />
*** [https://www.postgresql.org/message-id/618196db.1c69fb81.51c9a.00c1%40mx.google.com What about log_lock_waits, log_recovery_conflict_waits?]<br />
** [https://www.postgresql.org/message-id/20220216212952.GH31460@telsasoft.com initdb: default_toast_compression=lz4]<br />
** recovery_prefetch<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br />
** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer]<br />
** [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report]<br />
** [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report]<br />
<br />
* [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and >MaxAllocSize lengths]<br />
** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br />
** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br />
** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/f8a4105f076544c180a87ef0c4822352%40stmuk.bayern.de Extension pg_trgm, permissions and pg_dump order]<br />
** Affects all stable branches.<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-Wzn22s42h4Lh6v96GsXSKGd%3D_6b76mjqip_WFCGnBmTJCw%40mail.gmail.com CLUSTER sort on abbreviated expressions is broken]<br />
** Affects all stable branches.<br />
** Fixed by: {{PgCommitURL|8ab0ebb9a842dc6063d1374a38b47a3b7ee64afe}}<br />
<br />
* [https://www.postgresql.org/message-id/17485-396609c6925b982d%40postgresql.org Records missing from Primary Key index when doing REINDEX INDEX CONCURRENTLY]<br />
** Affects v14<br />
** Fixed by: {{PgCommitURL|e28bb885196916b0a3d898ae4f2be0e38108d81b}}<br />
<br />
== Non-bugs ==<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 15beta2 ===<br />
<br />
* [https://www.postgresql.org/message-id/CA+HiwqGAGobiiHR8nH382HJxqm1mzZs8=3oKPXnXivWoFSZmNA@mail.gmail.com pgbench --partitions=0]<br />
** Owner; Michael Paquier (6f164e6d17616a157ea5d9e34dbb1b211c080c41)<br />
** Fixed by: {{PgCommitURL|27f1366050c6cd8c1ea5f03b367a5a167ebf34b7}}<br />
* [https://www.postgresql.org/message-id/3813350.1652111765%40sss.pgh.pa.us psql now shows zero elapsed time after an error]<br />
** Owner: Peter Eisentraut<br />
** Fixed by: {{PgCommitURL|9520f8d92a8681e441cc863422babd544353dd39}}<br />
* [https://www.postgresql.org/message-id/17495-7ffe2fa0b261b9fa@postgresql.org Regression in 15beta1 when filtering subquery including row_number window function]<br />
** Owner: David Rowley (9d9c02ccd1aea8e9131d8f4edb21bf1687e40782)<br />
** Fixed by: {{PgCommitURL|3e9abd2eb1b1f6863250f060290f514f30ce8044}}<br />
* [https://www.postgresql.org/message-id/20220524235250.gtt3uu5zktfkr4hv@alap3.anarazel.de Safety of subtrans ID caching]<br />
** Owner: Michael Paquier (06f5295af673df795e8e70e28c43d61c2817b6df)<br />
** Fixed by: {{PgCommitURL|b4529005fd387e863bfa9eb863629b1183c0449c}}<br />
* [https://www.postgresql.org/message-id/f80ace33-11fb-1cd3-20f8-98f51d151088@enterprisedb.com pg_upgrade test writes to source directory]<br />
** Owner: Michael Paquier (322becb6085cb92d3708635eea61b45776bf27b6)<br />
** Fixed by: {{PgCommitURL|15b6d2155375dee2fcba072fffa03c1c8b44656c}}<br />
<br />
=== resolved before 15beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/de57761c-b99b-3435-b0a6-474c72b1149a%40enterprisedb.com libpq: duplicate error message after connection loss]<br />
** Fixed by: {{PgCommitURL|93909599cdba64c8759d646983c0a4ef93de1e50}}<br />
<br />
* [https://www.postgresql.org/message-id/fab3b90a-914d-46a9-beb0-df011ee39ee5%40www.fastmail.com MERGE: ERROR: variable not found in subplan target lists]<br />
** Fixed by: {{PgCommitURL|ce4f46fdc814eb1b704d81640f6d8f03625d0f53}}<br />
<br />
* [https://www.postgresql.org/message-id/20220212211316.GK31460%40telsasoft.com Buildfarm warnings]<br />
** pg_basebackup.c:1261:35: warning: storing the address of local variable archive_filename in progress_filename [-Wdangling-pointer=]<br />
** new in 23a1c6578 - looks like a real error @Robert Haas<br />
** Fixed at: {{PgCommitURL|62cb7427d1e491faf8612a82c2e3711a8cd65422}}<br />
<br />
* [https://www.postgresql.org/message-id/20220311010223.GI28503@telsasoft.com pg_basebackup serverside compression broken with stdout and manifests]<br />
** Fixed at: {{PgCommitURL|b2de45f9200d9adcac50015521574696dc464ccd}}<br />
<br />
* pg_basebackup: bbstreamer_lz4.c:172: bbstreamer_lz4_compressor_content: Assertion `mystreamer->base.bbs_buffer.maxlen >= out_bound' failed. <br />
** [https://www.postgresql.org/message-id/20220316151253.GB28503@telsasoft.com basebackup LZ4 to stdout]<br />
** Owner: Robert Haas (dab298471ff2f91f33bc25bfb73e435d3ab02148)<br />
** Fixed at: {{PgCommitURL|afb529e6772b4e2b065644a2204697eeaf6c9a96}}<br />
<br />
* [https://www.postgresql.org/message-id/CAKFQuwamFuaQHKdhcMt4Gbw5+Hca2UE741B8gOOXoA=TtAd2Yw@mail.gmail.com Incorrect reset timestamp in stats after crash recovery]<br />
** Owner: Andres Freund (5891c7a8ed8f2d3d577e7eea34dacff12d7b6bbd)<br />
** Fixed at: {{PgCommitURL|5cd1c40b3ce9600f129fd1fea9850e1affaf31d5}}<br />
<br />
* [https://www.postgresql.org/message-id/YlPQGNAAa04raObK@paquier.xyz Fixes for compression options of pg_receivewal and refactoring of backup_compression.{c,h}]<br />
** Owner: Michael Paquier (babbbb595d2322da095a1e6703171b3f1f2815cb)<br />
** Fixed at: {{PgCommitURL|042a923ad53dfbe39a9d5012d6c3cf3c9c338884}}<br />
<br />
* [https://www.postgresql.org/message-id/CA+TgmoazKcKUWtqVa0xZqSzbKgTH+X-aw4V7GyLD68EpDLMh8A@mail.gmail.com Remove compatibility from pg_basebackup?]<br />
** Fixed at: {{PgCommitURL|9cd28c2e5f11dfeef64a14035b82e70acead65fd}}<br />
<br />
* [https://www.postgresql.org/message-id/4015413.1649454951%40sss.pgh.pa.us Timing-dependent failure in 002_archiving.pl]<br />
** Owner: Michael Paquier (46dea2419ee7895a4eb3d048317682e6f18a17e1)<br />
** Fixed at: {{PgCommitURL|e61efafcb82c605dcc78f668685223e20d2f7ad8}}, {{PgCommitURL|1a8b110539efe18803c1fa8aa452a2178dbad9a9}}<br />
<br />
* [https://www.postgresql.org/message-id/CA+hUKGJRbzaAOUtBUcjF5hLtaSHnJUqXmtiaLEoi53zeWSizeA@mail.gmail.com qsort performance regression]<br />
** Owner: John Naylor (6974924347c908335607a4a2f252213d58e21b7c)<br />
** Fixed at: {{PgCommitURL|99c754129d787ea4ce3b34b9f4c5f5e74c45ab6a}}<br />
<br />
* [https://www.postgresql.org/message-id/YlZyp26LVVfmwfgW@paquier.xyz Small issues with CLUSTER on partitioned tables]<br />
** Owner: Alvaro Herrera (cfdd03f45e6afc632fbe70519250ec19167d6765)<br />
** Fixed at: {{PgCommitURL|3f19e176ae0f55a653d62e1504dbe5ad8c1006a0}}, {{PgCommitURL|21a10368eb3fce73f146d7e48b4d81496f60d965}}<br />
<br />
* [https://www.postgresql.org/message-id/20220408124338.GK24419@telsasoft.com asynchronous execution crash in trivial_subqueryscan()]<br />
** Owner: Etsuro Fujita (c2bb02bc2e858ba345b8b33f1f3a54628f719d93)<br />
** Fixed at: {{PgCommitURL|5c854e7a2c8a6cd26040e0f9949e7a4a007f6366}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/20220209220004.kb3dgtn2x2k2gtdm%40alap3.anarazel.de Corruption due to relfilenode reuse]<br />
** pg_upgrade can corrupt data with the new OIDs preservation feature<br />
*** Fixed at: {{PgCommitURL|e2f65f42555ff531c6d7c8f151526b4ef7c016f8}}<br />
** the ProcSignalBarrier solution this builds on also turns out to have a small race/hole<br />
*** Fixed at: {{PgCommitURL|b74e94dc27fdbb13954f230b1d1298430afa6c0c}}<br />
** Owner: Thomas Munro, Robert Haas<br />
<br />
* [https://www.postgresql.org/message-id/20220502042718.GB1565149@rfd.leadboat.com Some issues with the TAP tests of pg_upgrade]<br />
** Owner: Michael Paquier<br />
** Fixed at: {{PgCommitURL|7dd3ee508432730d15c5d3032f37362f6b6e4dd8}}<br />
<br />
* [https://www.postgresql.org/message-id/CAMbWs4-LN%3DbF8f9eU2R94dJtF54DfDvBq%2BovqHnOQqbinYDrUw%40mail.gmail.com Crash in _outPathTarget]<br />
** Owner: Peter Eisentraut<br />
** Fixed at: {{PgCommitURL|9ddf251f94090cebf1bd8fc18396cb8a4b580d04}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/Ymd/e5eeZMNAkrXo%40paquier.xyz#23885a148c6899cc874a7bf68f228777 Instability of regression test of pg_walinspect]<br />
** Owner: Jeff Davis<br />
** Fixed at: {{PgCommitURL|ed57cac84d1c5642737dab1e4c4b8cb4f0c4305f}}<br />
<br />
* [https://www.postgresql.org/message-id/YkfeMNYRCGhySKyg%40ahch-to crash with JSON constructors and window functions]<br />
** Owner: Andrew Dunstan (f4fb45d15c59d7add2e1b81a9d477d0119a9691a)<br />
** Fixed at: {{PgCommitURL|4eb9798879680dcc0e3ebb301cf6f925dfa69422}}, {{PgCommitURL|112fdb3528465cc14a2f1dff3dc27f100326d885}}<br />
<br />
* [https://www.postgresql.org/message-id/CAA4eK1LpBFU49Ohbnk%3Ddv_v9YP%2BKqh1%2BSf8i%2B%2B_s-QhD1Gy4Qw%40mail.gmail.com 013_partition.pl failing]<br />
** Fixed at: {{PgCommitURL|dd4ab6fd6528e160571986fa8817cee9f2645aa8}}<br />
<br />
* [https://www.postgresql.org/message-id/Yni6ZHkGotUU+RSf@paquier.xyz Avoid garbage logs with postgres -C on runtime-computed GUCs]<br />
** Fixed at: {{PgCommitURL|8bbf8461a3a2a38ce5f2952a025385b6938a61f7}}<br />
** Owner: Michael Paquier<br />
<br />
* [https://www.postgresql.org/message-id/20220506234924.6mxxotl3xl63db3l@alap3.anarazel.de Some issues with mark_pgdllimport.pl]<br />
** Fixed at: {{PgCommitURL|5edeb574285ecbcc47f0b769a7e363404db0155b}}<br />
** Owner: Robert Haas<br />
<br />
* [https://www.postgresql.org/message-id/1656446.1650043715%40sss.pgh.pa.us Crash in new pgstats code]<br />
** Initially reported issue was fixed by {{PgCommitURL|4a736a161c306fcfed970e6b649f2f03f465ac24}}, but there may be more to do here.<br />
** Owner: Andres Freund<br />
<br />
* [https://www.postgresql.org/message-id/b3463b8c-2328-dcac-0136-af95715493c1%40xs4all.nl TRAP: FailedAssertion("tabstat->trans == trans", File: "pgstat_relation.c", Line: 508]<br />
** Fixed at: {{PgCommitURL|0cf16cb8ca4853b084c40eca310c4c9c3ebf7e2a}}<br />
** Owner: Andres Freund<br />
<br />
* [https://www.postgresql.org/message-id/YlGJGiofZiWN3elx@jrouhaud limitations of GetMaxBackends()]<br />
** Fixed at: {{PgCommitURL|4f2400cb3f10aa79f99fba680c198237da28dd38}}, {{PgCommitURL|ab02d702ef08343fba30d90fdf7df5950063e8c9}}, {{PgCommitURL|7fc0e7de9fb8306e84d1c15211aba4308f694455}}<br />
** Owner: Robert Haas (aa64f23b02924724eafbd9eadbf26d85df30a12b, and 4567596316d186c6e61c72df013797266fcac2f7)<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Feature Freeze: April 7, 2022 ('''Last Day to Commit Features''')<br />
* Beta 1: May 19, 2022<br />
* Beta 2: TBD<br />
* GA: TBD<br />
<br />
== See also ==<br />
<br />
* [[Release Management Team]]<br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Continuous_Integration&diff=36687Continuous Integration2022-01-17T13:39:42Z<p>Rjuju: </p>
<hr />
<div>Here is some information about how to use a variety of free continuous integration services, for PostgreSQL hackers. You'll need a BitBucket, GitHub, GitLab or other public source repository account. Then you'll need to select one or more CI service and tell it to watch your account. In the case of the popular repo hosts you don't have to create a new account on the CI provider websites -- you just grant access. Finally you'll have to tell it how to build your branches with one or more control file in your source tree. You can add the control files in an extra commit that lives only in your feature development branch. The PostgreSQL master branch doesn't have any CI control files in it.<br />
<br />
The unofficial PostgreSQL [http://cfbot.cputube.org/ Patch Tester] uses a couple of these services to test patches posted to the -hackers mailing list, and the branches it creates contain some example control files that might be useful.<br />
<br />
== Postgres CI on github ==<br />
<br />
This is the recommended way to test a patch on various environmment, using the same CI than the unofficial commit fest bot. You can configure it on your own github repository following the instruction at https://github.com/postgres/postgres/blob/master/src/tools/ci/README. It's very easy and can be configured in a few minutes. <br />
<br />
== AppVeyor ==<br />
<br />
[https://www.appveyor.com/ AppVeyor] builds and tests code on Windows, Linux, macOS. Support for GitHub, GitHub Enterprise, Bitbucket, GitLab, VSTS, Kiln or custom publicly accessible repos. [https://www.postgresql.org/message-id/flat/CAEepm%3D16YRw9knaXDBfs-aK%3DZNNKWAeZW%2BKvoSCQJm8uRqfUjg%40mail.gmail.com Discussion] [https://www.postgresql.org/message-id/flat/d8e78714-dc77-4a64-783f-e863ba4d951f%402ndquadrant.com MingGW, Cygwin, MSVC appveyor.yml files from Peter Eisentraut]<br />
<br />
== Cirrus CI ==<br />
<br />
[https://cirrus-ci.com/ Cirrus CI] supports at least Windows, Linux, FreeBSD and macOS, so it currently has the widest range of operating systems targeted by PostgreSQL (and there may [https://twitter.com/cirrus_labs/status/1278410390377021449 more soon]). It supports only Github as a source, and can be [https://cirrus-ci.org/guide/quick-start/ enabled for your account very easily from Github Marketplace]. It's free for open source projects.<br />
<br />
[https://github.com/macdice/postgres/tree/cirrus-ci Here] is a work-in-progress example of how to use it to build a PostgreSQL branch on those four operating systems. Click the green checkmark to see results, and look at the top commit in that branch for the control files. The [https://github.com/macdice/postgres/blob/cirrus-ci/.cirrus.yml .cirrus.yml] file contains a list of to-do items for further work...<br />
<br />
== CodeCov ==<br />
<br />
[https://codecov.io CodeCov] is not a CI system, but is closely related and worth mentioning. If you configure Travis CI to build and test your branches with coverage enabled, you might also be interested to see code coverage information in a nice web interface on CodeCov.<br />
<br />
== Travis CI ==<br />
<br />
[https://travis-ci.org/ Travis CI] supports Linux, macOS and maybe more. Works only with GitHub. Controlled by a .travis.yml file.</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_14_Open_Items&diff=36282PostgreSQL 14 Open Items2021-08-08T06:02:16Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
* [https://www.postgresql.org/message-id/TYAPR01MB5866BA57688DF2770E2F95C6F5069@TYAPR01MB5866.jpnprd01.prod.outlook.com DECLARE STATEMENT and DEALLOCATE/DESCRIBE]<br />
** Owner: Michael Meskes<br />
<br />
* [https://www.postgresql.org/message-id/b5146fb1-ad9e-7d6e-f980-98ed68744a7c%40amazon.com Logical Decoding of relation rewrite with toast does not reset toast_hash]<br />
** Owner: Amit Kapila<br />
<br />
* [https://www.postgresql.org/message-id/20210730010355.6yodvn2ag3arfihi@alap3.anarazel.de Issues around autovacuum for partitioned tables]<br />
** Owner: Alvaro Herrera<br />
<br />
* [https://www.postgresql.org/message-id/0203588E-E609-43AF-9F4F-902854231EE7@enterprisedb.com Crash in regexp with {0}]<br />
** Owner: Tom Lane<br />
<br />
* [https://www.postgresql.org/message-id/OS0PR01MB5716935D4C2CC85A6143073F94EF9@OS0PR01MB5716.jpnprd01.prod.outlook.com wrong refresh when ALTER SUBSCRIPTION ADD/DROP PUBLICATION]<br />
** Owner: Peter Eisentraut<br />
<br />
* [https://www.postgresql.org/message-id/flat/20210807234407.icku2rnqyapsb3io%40alap3.anarazel.de elog.c query_id support vs shutdown]<br />
** Owner: Bruce Momjian<br />
<br />
== Decisions to Recheck Mid-Beta ==<br />
<br />
* [https://www.postgresql.org/message-id/E1lS8LX-0005sr-FZ%40gemulon.postgresql.org default setting of enable_memoize]<br />
** Owner: David Rowley<br />
<br />
* [https://www.postgresql.org/message-id/4170264.1620321747%40sss.pgh.pa.us Should we undo libpq change that leaves PQerrorMessage() nonempty after successful connect?]<br />
** Owner: Tom Lane<br />
<br />
* [https://www.postgresql.org/message-id/CABNQVagu3bZGqiTjb31a8D5Od3fUMs7Oh3gmZMQZVHZ=uWWWfQ@mail.gmail.com Consider back-patching typmod casting behavior change to stable branches]<br />
** Fixed in HEAD/v14 at: {{PgCommitURL|5c056b0c2519e602c2e98bace5b16d2ecde6454b}}<br />
** Owner: Tom Lane<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201016135230.GA23633%40alvherre.pgsql CREATE TABLE .. PARTITION OF fails to preserve tgenabled for inherited row triggers]<br />
** tgenabled lost on CREATE TABLE .. PARTITION OF, and on pg_dump, and comments on child triggers lost during pg_dump;<br />
** Those are resolved by f0e21f2f6 and df80fa2ee, but there's another issue with psql \d of non-inherited triggers<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer]<br />
** [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report]<br />
** [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report]<br />
<br />
* [https://www.postgresql.org/message-id/CAEudQAoR5e7=uMZ0otzuCVb25zTC8QQBe+2Dt1JRsa3u+XuwJg@mail.gmail.com could not rename temporary statistics file on Windows]<br />
** See {{PgCommitURL|909b449e00fc2f71e1a38569bbddbb6457d28485}} that has fixed a similar symptom for WAL segments. Most reporters of the WAL segment problem complained about this renaming issue as well.<br />
<br />
* [https://www.postgresql.org/message-id/20210422203603.fdnh3fu2mmfp2iov@alap3.anarazel.de Incorrect snapshot calculation when 2PC is in use]<br />
** Seems to be an old problem.<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/trinity-1c565d44-159f-488b-a518-caf13883134f-1611835701633%403c-app-gmx-bap78 hashagg broken by failing to spill grouping columns]<br />
** Fixed at: {{PgCommitURL|0ff865fbe50e82f17df8a9280fa01faf270b7f3f}}<br />
<br />
* [https://www.postgresql.org/message-id/CAE-ML+_EjH_fzfq1F3RJ1=XaaNG=-Jz-i3JqkNhXiLAsM3z-Ew@mail.gmail.com PITR promote bug: Checkpointer writes to older timeline]<br />
** Fixed at: {{PgCommitURL|595b9cba2ab0cdd057e02d3c23f34a8bcfd90a2d}}<br />
<br />
* [https://www.postgresql.org/message-id/YFBcRbnBiPdGZvfW%40paquier.xyz Permission failures with WAL files in 13~ on Windows]<br />
** Fixed at: {{PgCommitURL|78c24e97dd189f62187a99ef84016d0eb35a7978}}<br />
<br />
* [https://www.postgresql.org/message-id/CANiYTQsU7yMFpQYnv=BrcRVqK_3U3mtAzAsJCaqtzsDHfsUbdQ@mail.gmail.com CLOBBER_CACHE Server crashed with segfault 11 while executing clusterdb]<br />
** Fixed at: {{PgCommitURL|9d523119fd38fd205cb9c8ea8e7cceeb54355818}}<br />
<br />
* [https://www.postgresql.org/message-id/CAAV6ZkQRCVBh8qAY+SZiHnz+U+FqAGBBDaDTjF2yiKa2nJSLKg@mail.gmail.com Reference leak with tupledescs in plpgsql simple expressions]<br />
** Fixed at: {{PgCommitURL|c2db458c1036efae503ce5e451f8369e64c99541}}<br />
<br />
* [https://www.postgresql.org/message-id/a3be61d9-f44b-7fce-3dc8-d700fdfb6f48%402ndquadrant.com extract(julian) is undocumented and gives wrong result]<br />
** Fixed by documentation change at: {{PgCommitURL|79a5928ebcb726b7061bf265b5c6990e835e8c4f}}<br />
<br />
* [https://www.postgresql.org/message-id/CAGRY4nwxKUS_RvXFW-ugrZBYxPFFM5kjwKT5O+0+Stuga5b4+Q@mail.gmail.com lwlock dtrace probes do unnecessary work if dtrace is compiled in but disabled]<br />
** Fixed at: {{PgCommitURL|b94409a02f6122d77b5154e481c0819fed6b4c95}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/15990-eee2ac466b11293d%40postgresql.org Detoast failures after commit/rollback in plpgsql]<br />
** Fixed at: {{PgCommitURL|f21fadafaf0fb5ea4c9622d915972651273d62ce}} and {{PgCommitURL|84f5c2908dad81e8622b0406beea580e40bb03ac}}<br />
<br />
* [https://www.postgresql.org/message-id/3382681.1621381328%40sss.pgh.pa.us Subscription tests fail under CLOBBER_CACHE_ALWAYS]<br />
** Fixed at: {{PgCommitURL|b39630fd41f25b414d0ea9b30804f4105f2a0aff}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/534fca83789c4a378c7de379e9067d4f%40politie.nl XX000: unknown type of jsonb container.]<br />
** Fixed at: {{PgCommitURL|6ee41a301e70fc8e4ad383bad22d695f66ccb0ac}}<br />
<br />
* [https://www.postgresql.org/message-id/1884374.1617898865%40sss.pgh.pa.us Buildfarm does not test pg_stat_statements]<br />
** Fixed by buildfarm client change<br />
<br />
* [https://www.postgresql.org/message-id/17064-bb0d7904ef72add3%40postgresql.org Parallel VACUUM operations cause the error "global/pg_filenode.map contains incorrect checksum"]<br />
** Fixed at: {{PgCommitURL|b6d8d207}} and {{PgCommitURL|9b8ed0f52}}<br />
<br />
* [https://www.postgresql.org/message-id/378885e4-f85f-fc28-6c91-c4d1c080bf26%40amazon.com Assertion failure in HEAD and 13 after calling COMMIT in a stored proc]<br />
** Fixed at: {{PgCommitURL|d102aafb6259a6a412803d4b1d8c4f00aa17f67e}}<br />
<br />
* [https://www.postgresql.org/message-id/4aa370cb91ecf2f9885d98b80ad1109c%40postgrespro.ru Add PortalDrop in exec_execute_message]<br />
** Fixed at: {{PgCommitURL|bb4aed46a}} and {{PgCommitURL|4efcf47053}}<br />
<br />
* [https://www.postgresql.org/message-id/2591376.1621196582%40sss.pgh.pa.us snapshot-scalability logic fails after pg_upgrade, due to pg_resetwal issue]<br />
** Now seems likely that this is an old issue affecting every release, and that the snapshot-scalability work is not at fault<br />
** [https://commitfest.postgresql.org/33/3105/ Pending fix for pg_upgrade + pg_resetwal]<br />
** Fixed at: {{PgCommitURL|74cf7d46a91d601e0f8d957a7edbaeeb7df83efc}}<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
* [https://www.postgresql.org/message-id/20210216064214.GI28165%40telsasoft.com progress reporting for partitioned REINDEX]<br />
* [https://www.postgresql.org/message-id/YFnWBYinNf1s0Y6v@msg.df7cb.de pg_regress and tablespace removal]<br />
** [https://www.postgresql.org/message-id/YG/tf6HTZFj4hWlb@paquier.xyz Some patch]<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 14beta3 ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/20210530172418.GO2082%40telsasoft.com#d6544e507234cc76b9bc0a50026cd74b \dX doesn't check pg_statistics_obj_is_visible()]<br />
** Fixed at: {{PgCommitURL|f68b609230689f9886a46e5d9ab8d6cdd947e0dc}}<br />
<br />
* [https://www.postgresql.org/message-id/e1b4f05d-54ec-4f51-832b-c18cf5a161c0@www.fastmail.com remove_temp_files_after_crash should be a DEVELOPER GUC]<br />
** Fixed at: {{PgCommitURL|797b0fc0b078c7b4c46ef9f2d9e47aa2d98c6c63}}<br />
<br />
* [https://www.postgresql.org/message-id/20210526001359.GE3676@telsasoft.com recovery_init_sync_method should be PGC_SIGHUP?]<br />
** Fixed at: {{PgCommitURL|34a8b64b4e5f0cd818e5cc7f98846de57938ea57}}<br />
<br />
* [https://www.postgresql.org/message-id/YNZ2mnsbDVJQrA/a@paquier.xyz OOM on palloc() when parsing service file would cause libpq to exit() without reporting a failure]<br />
** Fixed at: {{PgCommitURL|8ec00dc5cd70e0e579e9fbf8661bc46f5ccd8078}}<br />
** Additional defenses added at: {{PgCommitURL|dc227eb82ea8bf6919cd81a182a084589ddce7f3}}<br />
<br />
* [https://www.postgresql.org/message-id/17076-89a16ae835d329b9%40postgresql.org incorrect code for reporting the hash partition associated with a particular modulus]<br />
** Fixed at: {{PgCommitURL|dd2364ced98553e0217bfe8f621cd4b0970db74a}}<br />
<br />
* [https://www.postgresql.org/message-id/c5269c65-f967-77c5-ff7c-15e621c47f6a%40gmail.com Bug in multirange selectivity estimation]<br />
** Fixed at: {{PgCommitURL|322e82b77ef4acb9697c6e4259292f5671cb85bb}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/704fb6fb99ec9864a4dbeda2478337d2%40postgrespro.ru autoanalyze of partitioned table causes it to lose its relhasindex]<br />
** Fixed at: {{PgCommitURL|d700518d744e53994fdded14b23ebc15b031b0dd}}<br />
<br />
* [https://www.postgresql.org/message-id/CAF7igB1r6wRfSCEAB-iZBKxkowWY6+dFF2jObSdd9+iVK+vHJg@mail.gmail.com Incorrect time maths in pgbench] and [https://www.postgresql.org/message-id/CAHLJuCW_8Vpcr0=t6O_gozrg3wXXWXZXDioYJd3NhvKriqgpfQ@mail.gmail.com second thread]<br />
** Fixed at: {{PgCommitURL|0e39a608ed5545cc6b9d538ac937c3c1ee8cdc36}}<br />
<br />
* [https://www.postgresql.org/message-id/60258efe-bd7e-4886-82e1-196e0cac5433%40postgresql.org unnesting multirange data types]<br />
** Fixed at: {{PgCommitURL|244ad5415557812a6ac4dc5d6e2ae908361d82c3}}<br />
<br />
* [https://www.postgresql.org/message-id/17066-16a37f6223a8470b@postgresql.org Cache lookup failed when null (unknown) is passed as anycompatiblemultirange]<br />
** Fixed at: {{PgCommitURL|336ea6e6ff1109e7b83370565e3cb211804fda0c}}<br />
<br />
* [https://www.postgresql.org/message-id/530153.1627425648%40sss.pgh.pa.us Degraded out-of-memory handling in libpq]<br />
** Fixed at: {{PgCommitURL|514b4c11d24701d2cc90ad75ed787bf1380af673}}<br />
<br />
=== resolved before 14beta2 ===<br />
<br />
* [https://www.postgresql.org/message-id/20210609184506.rqm5rikoikm47csf%40alap3.anarazel.de Snapshot scalability OldestXmin issue (can cause infinite loop during system catalog VACUUM)]<br />
** Fixed at: {{PgCommitURL|5a1e1d83022b976ebdec5cfa8f255c4278b75b8e}}<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkCYR0U7zXqXo0CgFaFwUDz1WbKq8ngjzKi4+AQ5f-mYQ@mail.gmail.com Generalize INDEX_CLEANUP to allow the user to disable the optimization that has VACUUM skip indexes in marginal cases with very few LP_DEAD items/deletable TIDs.]<br />
** Fixed at: {{PgCommitURL|3499df0dee8c4ea51d264a674df5b5e31991319a}}<br />
<br />
* [https://www.postgresql.org/message-id/20210324232224.vrfiij2rxxwqqjjb@alap3.anarazel.de Questions about pg_stat_wal] also [https://www.postgresql.org/message-id/E3774ACD-7894-451E-9F13-71E097D10595@oss.nttdata.com]<br />
** Fixed at: {{PgCommitURL|d8735b8b4651f5ed50afc472e236a8e6120f07f2}}<br />
** Fixed at: {{PgCommitURL|d780d7c0882fe9a385102b292907baaceb505ed0}}<br />
<br />
* [https://www.postgresql.org/message-id/YKMO%2B2gD8R8I2O5b%40paquier.xyz pg_dumpall misses --no-toast-compression]<br />
** Fixed at: {{PgCommitURL|694da1983e9569b2a2f96cd786ead6b8dba31f1d}} <br />
<br />
* [https://www.postgresql.org/message-id/YKQnUoYV63GRJBDD%40msg.df7cb.de portability issue with pgbench's permute() function]<br />
** Fixed at: {{PgCommitURL|0f516d039d8023163e82fa51104052306068dd69}}<br />
<br />
* [https://www.postgresql.org/message-id/35457b09-36f8-add3-1d07-6034fa585ca8@oss.nttdata.com compute_query_id and pg_stat_statements]<br />
** Fixed at {{PgCommitURL|cafde58b33}} and {{PgCommitURL|354f32d01d}}<br />
<br />
* [https://www.postgresql.org/message-id/CAOxo6X+dy-V58iEPFgst8ahPKEU+38NZzUuc+a7wDBZd4TrHMQ@mail.gmail.com Result Cache works incorrectly with unique joins]<br />
** Fixed at {{PgCommitURL|9e215378d7fbb7d4615be917917c52f246cc6c61}}<br />
<br />
* [https://www.postgresql.org/message-id/20210517204803.iyk5wwvwgtjcmc5w%40alap3.anarazel.de Move pg_attribute.attcompression to earlier in struct for reduced size?]<br />
** Fixed at {{PgCommitURL|f5024d8d7b04de2f5f4742ab433cc38160354861}}<br />
<br />
* [https://www.postgresql.org/message-id/17030-5844aecae42fe223@postgresql.org EXPLAIN can suffer from cannot decompile join alias var in plan tree]<br />
** Fixed at {{PgCommitURL|cba5c70b956810c61b3778f7041f92fbb8065acb}}<br />
<br />
* [https://www.postgresql.org/message-id/20210521211929.pcehg6f23icwstdb@alap3.anarazel.de Memory leak when rewriting tuples with recompressed toast values]<br />
** Fixed at {{PgCommitURL|fb0f5f0172edf9f63c8f70ea9c1ec043b61c770e}}<br />
<br />
* [https://www.postgresql.org/message-id/626613.1621787110%40sss.pgh.pa.us Redefine pg_attribute.attcompression]<br />
** Fixed at {{PgCommitURL|e6241d8e030fbd2746b3ea3f44e728224298f35b}}<br />
<br />
* [https://www.postgresql.org/message-id/1665197.1622065382%40sss.pgh.pa.us Undo bump of FirstBootstrapObjectId]<br />
** Fixed at {{PgCommitURL|a4390abecf0f5152cff864e82b67e5f6c8489698}}<br />
<br />
* [https://www.postgresql.org/message-id/CABOikdN-_858zojYN-2tNcHiVTw-nhxPwoQS4quExeweQfG1Ug%40mail.gmail.com Assertion failure while streaming toasted data]<br />
** Fixed at {{PgCommitURL|6f4bdf81529fdaf6744875b0be99ecb9bfb3b7e0}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/7817fb9ebd6661cdf9b67dec6e129a78%40postgrespro.ru Join pushdown issue in postgres_fdw updates]<br />
** Fixed at {{PgCommitURL|f61db909dfb94f3411f8719916601a11a905b95e}}<br />
<br />
* [https://www.postgresql.org/message-id/CAD21AoA%3D%3Df2VSw3c-Cp_y%3DWLKHMKc1D6s7g3YWsCOvgaYPpJcg%40mail.gmail.com Performance degradation of REFRESH MATERIALIZED VIEW]<br />
** Fixed at {{PgCommitURL|8e03eb92e9ad54e2f1ed8b5a73617601f6262f81}}<br />
<br />
* [https://www.postgresql.org/message-id/CAPmGK16Q4B2_KY%2BJH7rb7wQbw54AUprp7TMekGTd2T1B62yysQ%40mail.gmail.com Rescan of async Appends is broken when do_exec_prune=false]<br />
** Fixed at {{PgCommitURL|f3baaf28a6da588987b94a05a725894805c3eae9}}<br />
<br />
* [https://www.postgresql.org/message-id/504c276ab6eee000bb23d571ea9b0ced4250774e.camel%40vmware.com libpq dumps core while making an SSL connection to a server specified by hostaddr]<br />
** Fixed at {{PgCommitURL|37e1cce4ddf0be362e3093cee55493aee41bc423}}<br />
<br />
* [https://www.postgresql.org/message-id/B4A3AF82-79ED-4F4C-A4E5-CD2622098972%40enterprisedb.com logical replication of truncate command with trigger causes Assert]<br />
** Fixed at {{PgCommitURL|3a09d75b4f6cabc8331e228b6988dbfcd9afdfbe}}<br />
<br />
* [https://www.postgresql.org/message-id/3742981.1621533210%40sss.pgh.pa.us Reconsider catalog representation and uniqueness rules for procedures with output-only arguments]<br />
** Fixed at {{PgCommitURL|e56bce5d43789cce95d099554ae9593ada92b3b7}}<br />
<br />
* [https://www.postgresql.org/message-id/20210527003144.xxqppojoiwurc2iz@alap3.anarazel.de Performance regression of VACUUM FULL with the addition of recompression path in tuple rewrite]<br />
** Fixed at {{PgCommitURL|dbab0c07e5ba1f19a991da2d72972a8fe9a41bda}}<br />
<br />
* [https://www.postgresql.org/message-id/20210525161458.GZ3676%40telsasoft.com Document incompatibility with aggregates using system functions using anycompatiblearray]<br />
** Fixed at {{PgCommitURL|25dfb5a831a1b8a83a8a68453b4bbe38a5ef737e}}<br />
<br />
=== resolved before 14beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/OS0PR01MB611340CBD300A7C4FD6B6101FB5F9@OS0PR01MB6113.jpnprd01.prod.outlook.com FailedAssertion reported in lazy_scan_heap() when running logical replication]<br />
** Fixed at: {{PgCommitURL|c9787385db47ba423d845b34d58e158551c6335d}}<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gCXDSmFs2c%3DR%2BVGgn7FiYcLCsEFEuDNNLGfoha%3DpBE_g%40mail.gmail.com Assertion fail with window function and nested partitioned tables]<br />
** [https://www.postgresql.org/message-id/87sg8tqhsl.fsf@aurora.ydns.eu Older report]<br />
** Fixed at: {{PgCommitURL|fb2d645dd53ff571572d830e830fc8c368063802}}<br />
<br />
* [https://www.postgresql.org/message-id/1df88660-6f08-cc6e-b7e2-f85296a2bdab@oss.nttdata.com Atomic initialization of waitStart done at backend startup]<br />
** Fixed at: {{PgCommitURL|f05ed5a5cfa55878baa77a1e39d68cb09793b477}}<br />
<br />
* [https://www.postgresql.org/message-id/20210117215940.GE8560%40telsasoft.com pg_collation_actual_version() ERROR: cache lookup failed for collation 123]<br />
** Fixed at: {{PgCommitURL|0fb0a0503bfc125764c8dba4f515058145dc7f8b}}<br />
<br />
* [https://www.postgresql.org/message-id/fd3ba610085f1ff54623478cf2f7adf5af193cbb.camel@vmware.com cryptohash: missing locking functions for OpenSSL <= 1.0.2?]<br />
** Fixed at: {{PgCommitURL|2c0cefcd18161549e9e8b103f46c0f65fca84d99}}<br />
<br />
* [https://www.postgresql.org/message-id/CAHut%2BPuPGGASnh2Dy37VYODKULVQo-5oE%3DShc6gwtRizDt%3D%3DcA%40mail.gmail.com pg_subscription - substream column?]<br />
** Fixed at: {{PgCommitURL|7efeb214ad832fa96ea950d0906b1d2b96316d15}}<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gcs0zGOp6JXU2mMVdthYhuQpFk%3DS3V8DOKT%3DLZC1L36Q%40mail.gmail.com TOAST compression method of index columns]<br />
** Fixed at: {{PgCommitURL|5db1fd7823a1a12e2bdad98abc8e102fd71ffbda}}<br />
<br />
* [https://www.postgresql.org/message-id/20210402235337.GA4082@ahch-to Crash with encoding conversion functions]<br />
** Fixed at: {{PgCommitURL|c4c393b3ec83ceb4b4d7f37cdd5302126377d069}}<br />
<br />
* [https://www.postgresql.org/message-id/CAApHDvpYT10-nkSp8xXe-nbO3jmoaRyRFHbzh-RWMfAJynqgpQ@mail.gmail.com Crash with extended stats on expressions]<br />
** Fixed at: {{PgCommitURL|518442c7f334f3b05ea28b7ef50f1b551cfcc23e}}<br />
<br />
* [https://postgr.es/m/CA+TgmobwnGawnxufvqLCrcTy4HRhMepFiXQLY8YpVD+PTuwagA@mail.gmail.com Update TOAST documentation for LZ4 compression]<br />
** Fixed at: {{PgCommitURL|e8c435a824e123f43067ce6f69d66f14cfb8815e}}<br />
<br />
* [https://www.postgresql.org/message-id/20210404220802.GA728316@rfd.leadboat.com Behavior of pg_dump --extension with schemas]<br />
** Fixed at: {{PgCommitURL|344487e2db03f3cec13685a839dbc8a0e2a36750}}<br />
<br />
* [https://www.postgresql.org/message-id/OSZPR01MB631017521EE6887ADC9492E8FD759@OSZPR01MB6310.jpnprd01.prod.outlook.com psql query cancellation is broken], as are [https://www.postgresql.org/message-id/2671235.1618154047%40sss.pgh.pa.us autocommit], and [https://www.postgresql.org/message-id/YHTYOFBHDuGaz2gy@paquier.xyz error reporting]<br />
** Reverted by: {{PgCommitURL|fae65629cec824738ee11bf60f757239906d64fa}}<br />
<br />
* On Windows, collation version lookup (sometimes?) fails for names like "English_United States.1252", but works for names like "en-US".<br />
** Fixed at: {{PgCommitURL|9f12a3b95dd56c897f1aa3d756d8fb419e84a187}} -- this commit tolerates failure so at least we don't raise an error, but unfortunately we have no version information<br />
** Fixed at: {{PgCommitURL|1bf946bd43e545b86e567588b791311fe4e36a8c}} -- this commit documents the limitation<br />
<br />
* [https://www.postgresql.org/message-id/1820954.1617860500@sss.pgh.pa.us Handling of querystring inconsistent for parallel execution of SQL function bodies]<br />
** Fixed at: {{PgCommitURL|1111b2668d89bfcb6f502789158b1233ab4217a6}}<br />
<br />
* [https://www.postgresql.org/message-id/YHPkU8hFi4no4NSw@paquier.xyz Problems around compute_query_id]<br />
** Fixed at: {{PgCommitURL|db01f797dd48f826c62e1b8eea70f11fe7ff3efc}}<br />
<br />
* [https://www.postgresql.org/message-id/OS0PR01MB611383FA0FE92EB9DE21946AFB769@OS0PR01MB6113.jpnprd01.prod.outlook.com Table reference leak in logical replication]<br />
** Fixed at: {{PgCommitURL|f3b141c482552a57866c72919007d6481cd59ee3}}<br />
<br />
* [https://www.postgresql.org/message-id/20210410184226.GY6592%40telsasoft.com DETACH PARTITION CONCURRENTLY: Avoid adding redundant constraint]<br />
** Fixed at: {{PgCommitURL|7b357cc6ae}}<br />
<br />
* [https://www.postgresql.org/message-id/CC3F964B-8FA1-4A23-9D3E-6EA00BBFF0EE@enterprisedb.com Issues in PostgresNode and older major versions with multi-install]<br />
** Fixed at {{PgCommitURL|95c3a1956ec9eac686c1b69b033dd79211b72343}} and {{PgCommitURL|4c4eaf3d19201c5e2d9efebc590903dfaba0d3e5}}<br />
<br />
* [https://www.postgresql.org/message-id/3269784.1617215412%40sss.pgh.pa.us DETACH PARTITION CONCURRENTLY tests fail under CLOBBER_CACHE_ALWAYS]<br />
** Fixed at: {{PgCommitURL|8aba9322511f}}<br />
<br />
* [https://www.postgresql.org/message-id/551ed8c1-f531-818b-664a-2cecdab99cd8@oss.nttdata.com TRUNCATE on foreign tables and ONLY clause]<br />
** Fixed at: {{PgCommitURL|8e9ea08bae93a754d5075b7bc9c0b2bc71958bfd}}<br />
<br />
* [https://www.postgresql.org/message-id/CAMkU=1zKGWEJdBbYKw7Tn7cJmYR_UjgdcXTPDqJj=dNwCETBCQ@mail.gmail.com handling of character continuation in psql broken by sql body patch]<br />
** Fixed at: {{PgCommitURL|d9a9f4b4b92ad39e3c4e6600dc61d5603ddd6e24}}<br />
<br />
* [https://www.postgresql.org/message-id/20210505210947.GA27406%40telsasoft.com cache lookup failed for statistics object 123]<br />
** Fixed at: {{PgCommitURL|8d4b311d2494ca592e30aed03b29854d864eb846}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/CAFj8pRCL_Rjw-MCR6J7VX9OF7MR6PA5K8qUbrMvprW_e-aHkfQ%40mail.gmail.com batch fdw insert bug]<br />
** Fixed at: {{PgCommitURL|c6a01d924939306e95c8deafd09352be6a955648}}<br />
<br />
* [https://www.postgresql.org/message-id/3564817.1618420687@sss.pgh.pa.us Bogus collation version recording in recordMultipleDependencies]<br />
** Fixed at: {{PgCommitURL|ec48314708262d8ea6cdcb83f803fc83dd89e721}} (Feature revert)<br />
<br />
* [https://www.postgresql.org/message-id/773932.1619022622@sss.pgh.pa.us Corruption issues with WAL prefetch?]<br />
** Fixed at: {{PgCommitURL|c2dc19342e05e081dc13b296787baa38352681ef}} (Feature revert)<br />
<br />
* [https://www.postgresql.org/message-id/YIetoZGq31L84v5d@paquier.xyz Small issues with CREATE TABLE COMPRESSION]<br />
** MSVC scripts don't support builds with lz4: fixed at {{PgCommitURL|9ca40dcd4d0cad43d95a9a253fafaa9a9ba7de24}}<br />
** pg_dump includes no tests with compression methods of attributes and --no-toast-compression: fixed at {{PgCommitURL|63db0ac3f9e6bae313da67f640c95c0045b7f0ee}}<br />
** Documentation missing for --with-lz4 in installation instructions: fixed at {{PgCommitURL|02a93e7ef9612788081ef07ea1bbd0a8cc99ae99}}<br />
<br />
* [https://www.postgresql.org/message-id/20210319185247.ldebgpdaxsowiflw@alap3.anarazel.de Replication slot stats misgivings]<br />
** Fixed at: {{PgCommitURL|3fa17d37716f978f80dfcdab4e7c73f3a24e7a48}}<br />
** Fixed at: {{PgCommitURL|592f00f8dec68038301467a904ac514eddabf6cd}}<br />
** Fixed at: {{PgCommitURL|cca57c1d9bf7eeba5b81115e0b82651cf3d8e4ea}}<br />
** Fixed at: {{PgCommitURL|f5fc2f5b23d1b1dff60f8ca5dc211161df47eda4}}<br />
<br />
* [https://www.postgresql.org/message-id/CAPmGK158e9sJOfuWxfn%2B0ynrspXQU3JhNjSCbaoeSzMvnga%2Bbw%40mail.gmail.com FDW: crash with DDL and async/batch option]<br />
** Fixed at: {{PgCommitURL|a784859f4480ceaa05a00ca35311071ca33483d1}}<br />
<br />
* [https://www.postgresql.org/message-id/20210409213155.GA23912%40alvherre.pgsql should autoanalyze for partitioned tables handle ATTACH/DETACH/DROP?]<br />
** Fixed at: {{PgCommitURL|1b5617eb844cd2470a334c1d2eec66cf9b39c41a}} (docs)<br />
<br />
* [https://www.postgresql.org/message-id/CALT9ZEE7OiszofHELnjPhX%3DhV92PiKn8haSZ4_FWBAw4diaRdQ%40mail.gmail.com OOM in spgist insert]<br />
** Fixed at: {{PgCommitURL|c3c35a733c77b298d3cf7e7de2eeb4aea540a631}}<br />
<br />
== Won't Fix ==<br />
<br />
* [https://www.postgresql.org/message-id/92408.1618772924%40sss.pgh.pa.us SQL-standard function body: pg_dump should handle circular dependencies]<br />
** Owner: Peter Eisentraut<br />
* [https://www.postgresql.org/message-id/17061-dd7f4825b7da3a9d%40postgresql.org SEARCH BREADTH FIRST produces a composite column whose fields can't be accessed]<br />
** Owner: Peter Eisentraut<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Feature Freeze: April 7, 2021 ('''Last Day to Commit Features''')<br />
* Beta 1: May 20, 2021<br />
* Beta 2: June 24, 2021<br />
* Beta 3: <br />
* RC 1: <br />
* GA: <br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_14_Open_Items&diff=35918PostgreSQL 14 Open Items2021-04-15T10:24:39Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
* [https://www.postgresql.org/message-id/CAD21AoA%3D%3Df2VSw3c-Cp_y%3DWLKHMKc1D6s7g3YWsCOvgaYPpJcg%40mail.gmail.com Performance degradation of REFRESH MATERIALIZED VIEW]<br />
<br />
* On Windows, collation version lookup (sometimes?) fails for names like "English_United States.1252", but works for names like "en-US" (BCP47 language tags). Unfortunately the default value chosen by initdb is in the former style (users would have to use something like initdb --lc-collate=en-US for the version reporting to work).<br />
** {{PgCommitURL|9f12a3b95dd56c897f1aa3d756d8fb419e84a187}} was committed to tolerate the failure so at least we don't raise an error, but unfortunately we have no version information<br />
** We should either figure out how to improve this situation (translate the names, but how? and when? ask the OS for a default in the BCP47 style? how?), or document the limitation<br />
<br />
* [https://www.postgresql.org/message-id/20210319185247.ldebgpdaxsowiflw@alap3.anarazel.de Replication slot stats misgivings]<br />
<br />
* [https://www.postgresql.org/message-id/CAApHDvpYT10-nkSp8xXe-nbO3jmoaRyRFHbzh-RWMfAJynqgpQ@mail.gmail.com Crash with extended stats on expressions]<br />
** Owner: Tomas Vondra<br />
<br />
* [https://www.postgresql.org/message-id/CC3F964B-8FA1-4A23-9D3E-6EA00BBFF0EE@enterprisedb.com Issues in PostgresNode and older major versions with multi-install]<br />
** Owner: Andrew Dunstan<br />
<br />
* [https://www.postgresql.org/message-id/1820954.1617860500@sss.pgh.pa.us Handling of querystring inconsistent for parallel execution of SQL function bodies]<br />
** Owner: Peter Eisentraut<br />
** Proposed patches at [https://www.postgresql.org/message-id/2197698.1617984583%40sss.pgh.pa.us]<br />
<br />
* [https://www.postgresql.org/message-id/OSZPR01MB631017521EE6887ADC9492E8FD759@OSZPR01MB6310.jpnprd01.prod.outlook.com psql query cancellation is broken], as are [https://www.postgresql.org/message-id/2671235.1618154047%40sss.pgh.pa.us autocommit], and [https://www.postgresql.org/message-id/YHTYOFBHDuGaz2gy@paquier.xyz error reporting]<br />
** Owner: Peter Eisentraut<br />
** Patch available at [https://www.postgresql.org/message-id/alpine.DEB.2.22.394.2104071516470.2724014@pseudo] for the query cancellation<br />
<br />
* [https://www.postgresql.org/message-id/3269784.1617215412%40sss.pgh.pa.us DETACH PARTITION CONCURRENTLY tests fail under CLOBBER_CACHE_ALWAYS]<br />
** Owner: Alvaro Herrera<br />
<br />
* [https://www.postgresql.org/message-id/OS0PR01MB611383FA0FE92EB9DE21946AFB769@OS0PR01MB6113.jpnprd01.prod.outlook.com Table reference leak in logical replication]<br />
** Owner: Heikki Linnakangas<br />
<br />
* [https://www.postgresql.org/message-id/20210409213155.GA23912%40alvherre.pgsql autoanalyze for partitioned tables should handle ATTACH/DETACH/DROP]<br />
** Owner: Alvaro Herrera<br />
<br />
* [https://www.postgresql.org/message-id/YHPkU8hFi4no4NSw@paquier.xyz Problems around compute_query_id]<br />
** Owner: Bruce Momjian<br />
<br />
* [https://www.postgresql.org/message-id/20210410184226.GY6592%40telsasoft.com DETACH PARTITION CONCURRENTLY: Avoid adding redundant constraint]<br />
** Owner: Alvaro Herrera<br />
<br />
* [https://www.postgresql.org/message-id/551ed8c1-f531-818b-664a-2cecdab99cd8@oss.nttdata.com TRUNCATE on foreign tables and ONLY clause]<br />
** Owner: Fujii Masao<br />
<br />
* [https://www.postgresql.org/message-id/20210324232224.vrfiij2rxxwqqjjb@alap3.anarazel.de Questions about pg_stat_wal]<br />
** Owner: Fujii Masao<br />
<br />
* [https://www.postgresql.org/message-id/3564817.1618420687@sss.pgh.pa.us Bogus collation version recording in recordMultipleDependencies]<br />
** Owner: Thomas Munro<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201016135230.GA23633%40alvherre.pgsql CREATE TABLE .. PARTITION OF fails to preserve tgenabled for inherited row triggers]<br />
** tgenabled lost on CREATE TABLE .. PARTITION OF, and on pg_dump, and comments on child triggers lost during pg_dump;<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer]<br />
** [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report]<br />
** [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report]<br />
<br />
* [https://www.postgresql.org/message-id/a3be61d9-f44b-7fce-3dc8-d700fdfb6f48%402ndquadrant.com extract(julian) is undocumented and gives wrong result]<br />
** With reimplementation of extract to return numeric, this might be an opportune time to fix this one way or the other.<br />
<br />
* [https://www.postgresql.org/message-id/CAGRY4nwxKUS_RvXFW-ugrZBYxPFFM5kjwKT5O+0+Stuga5b4+Q@mail.gmail.com lwlock dtrace probes do unnecessary work if dtrace is compiled in but disabled]<br />
** since PG13<br />
<br />
* [https://www.postgresql.org/message-id/1884374.1617898865%40sss.pgh.pa.us Buildfarm does not test pg_stat_statements]<br />
<br />
* [https://www.postgresql.org/message-id/CAEudQAoR5e7=uMZ0otzuCVb25zTC8QQBe+2Dt1JRsa3u+XuwJg@mail.gmail.com could not rename temporary statistics file on Windows]<br />
** See {{PgCommitURL|909b449e00fc2f71e1a38569bbddbb6457d28485}} that has fixed a similar symptom for WAL segments. Most reporters of the WAL segment problem complained about this renaming issue as well.<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/trinity-1c565d44-159f-488b-a518-caf13883134f-1611835701633%403c-app-gmx-bap78 hashagg broken by failing to spill grouping columns]<br />
** Fixed at: {{PgCommitURL|0ff865fbe50e82f17df8a9280fa01faf270b7f3f}}<br />
<br />
* [https://www.postgresql.org/message-id/CAE-ML+_EjH_fzfq1F3RJ1=XaaNG=-Jz-i3JqkNhXiLAsM3z-Ew@mail.gmail.com PITR promote bug: Checkpointer writes to older timeline]<br />
** Fixed at: {{PgCommitURL|595b9cba2ab0cdd057e02d3c23f34a8bcfd90a2d}}<br />
<br />
* [https://www.postgresql.org/message-id/YFBcRbnBiPdGZvfW%40paquier.xyz Permission failures with WAL files in 13~ on Windows]<br />
** Fixed at: {{PgCommitURL|78c24e97dd189f62187a99ef84016d0eb35a7978}}<br />
<br />
* [https://www.postgresql.org/message-id/CANiYTQsU7yMFpQYnv=BrcRVqK_3U3mtAzAsJCaqtzsDHfsUbdQ@mail.gmail.com CLOBBER_CACHE Server crashed with segfault 11 while executing clusterdb]<br />
** Fixed at: {{PgCommitURL|9d523119fd38fd205cb9c8ea8e7cceeb54355818}}<br />
<br />
* [https://www.postgresql.org/message-id/CAAV6ZkQRCVBh8qAY+SZiHnz+U+FqAGBBDaDTjF2yiKa2nJSLKg@mail.gmail.com Reference leak with tupledescs in plpgsql simple expressions]<br />
** Fixed at: {{PgCommitURL|c2db458c1036efae503ce5e451f8369e64c99541}}<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
* [https://www.postgresql.org/message-id/20210216064214.GI28165%40telsasoft.com progress reporting for partitioned REINDEX]<br />
* [https://www.postgresql.org/message-id/YFnWBYinNf1s0Y6v@msg.df7cb.de pg_regress and tablespace removal]<br />
** [https://www.postgresql.org/message-id/YG/tf6HTZFj4hWlb@paquier.xyz Some patch]<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 14beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gCXDSmFs2c%3DR%2BVGgn7FiYcLCsEFEuDNNLGfoha%3DpBE_g%40mail.gmail.com Assertion fail with window function and nested partitioned tables]<br />
** [https://www.postgresql.org/message-id/87sg8tqhsl.fsf@aurora.ydns.eu Older report]<br />
** Fixed at: {{PgCommitURL|fb2d645dd53ff571572d830e830fc8c368063802}}<br />
<br />
* [https://www.postgresql.org/message-id/1df88660-6f08-cc6e-b7e2-f85296a2bdab@oss.nttdata.com Atomic initialization of waitStart done at backend startup]<br />
** Fixed at: {{PgCommitURL|f05ed5a5cfa55878baa77a1e39d68cb09793b477}}<br />
<br />
* [https://www.postgresql.org/message-id/20210117215940.GE8560%40telsasoft.com pg_collation_actual_version() ERROR: cache lookup failed for collation 123]<br />
** Fixed at: {{PgCommitURL|0fb0a0503bfc125764c8dba4f515058145dc7f8b}}<br />
<br />
* [https://www.postgresql.org/message-id/fd3ba610085f1ff54623478cf2f7adf5af193cbb.camel@vmware.com cryptohash: missing locking functions for OpenSSL <= 1.0.2?]<br />
** Fixed at: {{PgCommitURL|2c0cefcd18161549e9e8b103f46c0f65fca84d99}}<br />
<br />
* [https://www.postgresql.org/message-id/CAHut%2BPuPGGASnh2Dy37VYODKULVQo-5oE%3DShc6gwtRizDt%3D%3DcA%40mail.gmail.com pg_subscription - substream column?]<br />
** Fixed at: {{PgCommitURL|7efeb214ad832fa96ea950d0906b1d2b96316d15}}<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gcs0zGOp6JXU2mMVdthYhuQpFk%3DS3V8DOKT%3DLZC1L36Q%40mail.gmail.com TOAST compression method of index columns]<br />
** Fixed at: {{PgCommitURL|5db1fd7823a1a12e2bdad98abc8e102fd71ffbda}}<br />
<br />
* [https://www.postgresql.org/message-id/20210402235337.GA4082@ahch-to Crash with encoding conversion functions]<br />
** Fixed at: {{PgCommitURL|c4c393b3ec83ceb4b4d7f37cdd5302126377d069}}<br />
<br />
* [https://postgr.es/m/CA+TgmobwnGawnxufvqLCrcTy4HRhMepFiXQLY8YpVD+PTuwagA@mail.gmail.com Update TOAST documentation for LZ4 compression]<br />
** Fixed at: {{PgCommitURL|e8c435a824e123f43067ce6f69d66f14cfb8815e}}<br />
<br />
* [https://www.postgresql.org/message-id/20210404220802.GA728316@rfd.leadboat.com Behavior of pg_dump --extension with schemas]<br />
** Fixed at: {{PgCommitURL|344487e2db03f3cec13685a839dbc8a0e2a36750}}<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Feature Freeze: April 7, 2021 ('''Last Day to Commit Features''')<br />
* Beta 1: <br />
* Beta 2: <br />
* Beta 3: <br />
* RC 1: <br />
* GA: <br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_14_Open_Items&diff=35915PostgreSQL 14 Open Items2021-04-15T04:14:27Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
* [https://www.postgresql.org/message-id/CAD21AoA%3D%3Df2VSw3c-Cp_y%3DWLKHMKc1D6s7g3YWsCOvgaYPpJcg%40mail.gmail.com Performance degradation of REFRESH MATERIALIZED VIEW]<br />
<br />
* On Windows, collation version lookup (sometimes?) fails for names like "English_United States.1252", but works for names like "en-US" (BCP47 language tags). Unfortunately the default value chosen by initdb is in the former style (users would have to use something like initdb --lc-collate=en-US for the version reporting to work).<br />
** {{PgCommitURL|9f12a3b95dd56c897f1aa3d756d8fb419e84a187}} was committed to tolerate the failure so at least we don't raise an error, but unfortunately we have no version information<br />
** We should either figure out how to improve this situation (translate the names, but how? and when? ask the OS for a default in the BCP47 style? how?), or document the limitation<br />
<br />
* [https://www.postgresql.org/message-id/20210319185247.ldebgpdaxsowiflw@alap3.anarazel.de Replication slot stats misgivings]<br />
<br />
* [https://www.postgresql.org/message-id/CAApHDvpYT10-nkSp8xXe-nbO3jmoaRyRFHbzh-RWMfAJynqgpQ@mail.gmail.com Crash with extended stats on expressions]<br />
** Owner: Tomas Vondra<br />
<br />
* [https://www.postgresql.org/message-id/CC3F964B-8FA1-4A23-9D3E-6EA00BBFF0EE@enterprisedb.com Issues in PostgresNode and older major versions with multi-install]<br />
** Owner: Andrew Dunstan<br />
<br />
* [https://www.postgresql.org/message-id/1820954.1617860500@sss.pgh.pa.us Handling of querystring inconsistent for parallel execution of SQL function bodies]<br />
** Owner: Peter Eisentraut<br />
** Proposed patches at [https://www.postgresql.org/message-id/2197698.1617984583%40sss.pgh.pa.us]<br />
<br />
* [https://www.postgresql.org/message-id/OSZPR01MB631017521EE6887ADC9492E8FD759@OSZPR01MB6310.jpnprd01.prod.outlook.com psql query cancellation is broken], as are [https://www.postgresql.org/message-id/2671235.1618154047%40sss.pgh.pa.us autocommit], and [https://www.postgresql.org/message-id/YHTYOFBHDuGaz2gy@paquier.xyz error reporting]<br />
** Owner: Peter Eisentraut<br />
** Patch available at [https://www.postgresql.org/message-id/alpine.DEB.2.22.394.2104071516470.2724014@pseudo] for the query cancellation<br />
<br />
* [https://www.postgresql.org/message-id/3269784.1617215412%40sss.pgh.pa.us DETACH PARTITION CONCURRENTLY tests fail under CLOBBER_CACHE_ALWAYS]<br />
** Owner: Alvaro Herrera<br />
<br />
* [https://www.postgresql.org/message-id/OS0PR01MB611383FA0FE92EB9DE21946AFB769@OS0PR01MB6113.jpnprd01.prod.outlook.com Table reference leak in logical replication]<br />
** Owner: Heikki Linnakangas<br />
<br />
* [https://www.postgresql.org/message-id/20210409213155.GA23912%40alvherre.pgsql autoanalyze for partitioned tables should handle ATTACH/DETACH/DROP]<br />
** Owner: Alvaro Herrera<br />
<br />
* [https://www.postgresql.org/message-id/YHPkU8hFi4no4NSw@paquier.xyz Problems around compute_query_id]<br />
** Owner: Bruce Momjian<br />
<br />
* [https://www.postgresql.org/message-id/20210410184226.GY6592%40telsasoft.com DETACH PARTITION CONCURRENTLY: Avoid adding redundant constraint]<br />
** Owner: Alvaro Herrera<br />
<br />
* [https://www.postgresql.org/message-id/551ed8c1-f531-818b-664a-2cecdab99cd8@oss.nttdata.com TRUNCATE on foreign tables and ONLY clause]<br />
** Owner: Fujii Masao<br />
<br />
* [https://www.postgresql.org/message-id/20210324232224.vrfiij2rxxwqqjjb@alap3.anarazel.de Questions about pg_stat_wal]<br />
** Owner: Fujii Masao<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201016135230.GA23633%40alvherre.pgsql CREATE TABLE .. PARTITION OF fails to preserve tgenabled for inherited row triggers]<br />
** tgenabled lost on CREATE TABLE .. PARTITION OF, and on pg_dump, and comments on child triggers lost during pg_dump;<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer]<br />
** [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report]<br />
** [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report]<br />
<br />
* [https://www.postgresql.org/message-id/a3be61d9-f44b-7fce-3dc8-d700fdfb6f48%402ndquadrant.com extract(julian) is undocumented and gives wrong result]<br />
** With reimplementation of extract to return numeric, this might be an opportune time to fix this one way or the other.<br />
<br />
* [https://www.postgresql.org/message-id/CAGRY4nwxKUS_RvXFW-ugrZBYxPFFM5kjwKT5O+0+Stuga5b4+Q@mail.gmail.com lwlock dtrace probes do unnecessary work if dtrace is compiled in but disabled]<br />
** since PG13<br />
<br />
* [https://www.postgresql.org/message-id/1884374.1617898865%40sss.pgh.pa.us Buildfarm does not test pg_stat_statements]<br />
<br />
* [https://www.postgresql.org/message-id/CAEudQAoR5e7=uMZ0otzuCVb25zTC8QQBe+2Dt1JRsa3u+XuwJg@mail.gmail.com could not rename temporary statistics file on Windows]<br />
** See {{PgCommitURL|909b449e00fc2f71e1a38569bbddbb6457d28485}} that has fixed a similar symptom for WAL segments. Most reporters of the WAL segment problem complained about this renaming issue as well.<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/trinity-1c565d44-159f-488b-a518-caf13883134f-1611835701633%403c-app-gmx-bap78 hashagg broken by failing to spill grouping columns]<br />
** Fixed at: {{PgCommitURL|0ff865fbe50e82f17df8a9280fa01faf270b7f3f}}<br />
<br />
* [https://www.postgresql.org/message-id/CAE-ML+_EjH_fzfq1F3RJ1=XaaNG=-Jz-i3JqkNhXiLAsM3z-Ew@mail.gmail.com PITR promote bug: Checkpointer writes to older timeline]<br />
** Fixed at: {{PgCommitURL|595b9cba2ab0cdd057e02d3c23f34a8bcfd90a2d}}<br />
<br />
* [https://www.postgresql.org/message-id/YFBcRbnBiPdGZvfW%40paquier.xyz Permission failures with WAL files in 13~ on Windows]<br />
** Fixed at: {{PgCommitURL|78c24e97dd189f62187a99ef84016d0eb35a7978}}<br />
<br />
* [https://www.postgresql.org/message-id/CANiYTQsU7yMFpQYnv=BrcRVqK_3U3mtAzAsJCaqtzsDHfsUbdQ@mail.gmail.com CLOBBER_CACHE Server crashed with segfault 11 while executing clusterdb]<br />
** Fixed at: {{PgCommitURL|9d523119fd38fd205cb9c8ea8e7cceeb54355818}}<br />
<br />
* [https://www.postgresql.org/message-id/CAAV6ZkQRCVBh8qAY+SZiHnz+U+FqAGBBDaDTjF2yiKa2nJSLKg@mail.gmail.com Reference leak with tupledescs in plpgsql simple expressions]<br />
** Fixed at: {{PgCommitURL|c2db458c1036efae503ce5e451f8369e64c99541}}<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
* [https://www.postgresql.org/message-id/20210216064214.GI28165%40telsasoft.com progress reporting for partitioned REINDEX]<br />
* [https://www.postgresql.org/message-id/YFnWBYinNf1s0Y6v@msg.df7cb.de pg_regress and tablespace removal]<br />
** [https://www.postgresql.org/message-id/YG/tf6HTZFj4hWlb@paquier.xyz Some patch]<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 14beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gCXDSmFs2c%3DR%2BVGgn7FiYcLCsEFEuDNNLGfoha%3DpBE_g%40mail.gmail.com Assertion fail with window function and nested partitioned tables]<br />
** [https://www.postgresql.org/message-id/87sg8tqhsl.fsf@aurora.ydns.eu Older report]<br />
** Fixed at: {{PgCommitURL|fb2d645dd53ff571572d830e830fc8c368063802}}<br />
<br />
* [https://www.postgresql.org/message-id/1df88660-6f08-cc6e-b7e2-f85296a2bdab@oss.nttdata.com Atomic initialization of waitStart done at backend startup]<br />
** Fixed at: {{PgCommitURL|f05ed5a5cfa55878baa77a1e39d68cb09793b477}}<br />
<br />
* [https://www.postgresql.org/message-id/20210117215940.GE8560%40telsasoft.com pg_collation_actual_version() ERROR: cache lookup failed for collation 123]<br />
** Fixed at: {{PgCommitURL|0fb0a0503bfc125764c8dba4f515058145dc7f8b}}<br />
<br />
* [https://www.postgresql.org/message-id/fd3ba610085f1ff54623478cf2f7adf5af193cbb.camel@vmware.com cryptohash: missing locking functions for OpenSSL <= 1.0.2?]<br />
** Fixed at: {{PgCommitURL|2c0cefcd18161549e9e8b103f46c0f65fca84d99}}<br />
<br />
* [https://www.postgresql.org/message-id/CAHut%2BPuPGGASnh2Dy37VYODKULVQo-5oE%3DShc6gwtRizDt%3D%3DcA%40mail.gmail.com pg_subscription - substream column?]<br />
** Fixed at: {{PgCommitURL|7efeb214ad832fa96ea950d0906b1d2b96316d15}}<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gcs0zGOp6JXU2mMVdthYhuQpFk%3DS3V8DOKT%3DLZC1L36Q%40mail.gmail.com TOAST compression method of index columns]<br />
** Fixed at: {{PgCommitURL|5db1fd7823a1a12e2bdad98abc8e102fd71ffbda}}<br />
<br />
* [https://www.postgresql.org/message-id/20210402235337.GA4082@ahch-to Crash with encoding conversion functions]<br />
** Fixed at: {{PgCommitURL|c4c393b3ec83ceb4b4d7f37cdd5302126377d069}}<br />
<br />
* [https://postgr.es/m/CA+TgmobwnGawnxufvqLCrcTy4HRhMepFiXQLY8YpVD+PTuwagA@mail.gmail.com Update TOAST documentation for LZ4 compression]<br />
** Fixed at: {{PgCommitURL|e8c435a824e123f43067ce6f69d66f14cfb8815e}}<br />
<br />
* [https://www.postgresql.org/message-id/20210404220802.GA728316@rfd.leadboat.com Behavior of pg_dump --extension with schemas]<br />
** Fixed at: {{PgCommitURL|344487e2db03f3cec13685a839dbc8a0e2a36750}}<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Feature Freeze: April 7, 2021 ('''Last Day to Commit Features''')<br />
* Beta 1: <br />
* Beta 2: <br />
* Beta 3: <br />
* RC 1: <br />
* GA: <br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_14_Open_Items&diff=35882PostgreSQL 14 Open Items2021-04-08T17:08:35Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
* [https://www.postgresql.org/message-id/CAD21AoA%3D%3Df2VSw3c-Cp_y%3DWLKHMKc1D6s7g3YWsCOvgaYPpJcg%40mail.gmail.com Performance degradation of REFRESH MATERIALIZED VIEW]<br />
<br />
* On Windows, collation version lookup (sometimes?) fails for names like "English_United States.1252", but works for names like "en-US" (BCP47 language tags). Unfortunately the default value chosen by initdb is in the former style (users would have to use something like initdb --lc-collate=en-US for the version reporting to work).<br />
** {{PgCommitURL|9f12a3b95dd56c897f1aa3d756d8fb419e84a187}} was committed to tolerate the failure so at least we don't raise an error, but unfortunately we have no version information<br />
** We should either figure out how to improve this situation (translate the names, but how? and when? ask the OS for a default in the BCP47 style? how?), or document the limitation<br />
<br />
* [https://www.postgresql.org/message-id/20210319185247.ldebgpdaxsowiflw@alap3.anarazel.de Replication slot stats misgivings]<br />
<br />
* [https://postgr.es/m/CA+TgmobwnGawnxufvqLCrcTy4HRhMepFiXQLY8YpVD+PTuwagA@mail.gmail.com Update TOAST documentation for LZ4 compression]<br />
** Owner: Robert Haas<br />
<br />
* [https://www.postgresql.org/message-id/YFnWBYinNf1s0Y6v@msg.df7cb.de pg_regress and tablespace removal]<br />
** Owner: Michael Paquier<br />
<br />
* [https://www.postgresql.org/message-id/CAApHDvpYT10-nkSp8xXe-nbO3jmoaRyRFHbzh-RWMfAJynqgpQ@mail.gmail.com Crash with extended stats on expressions]<br />
** Owner: Tomas Vondra<br />
<br />
* [https://www.postgresql.org/message-id/CC3F964B-8FA1-4A23-9D3E-6EA00BBFF0EE@enterprisedb.com Issues in PostgresNode and older major versions with multi-install]<br />
** Owner: Andrew Dunstan<br />
<br />
* [https://www.postgresql.org/message-id/20210404220802.GA728316@rfd.leadboat.com Behavior of pg_dump --extension with schemas]<br />
** Owner: Michael Paquier<br />
<br />
* [https://www.postgresql.org/message-id/CAAV6ZkQRCVBh8qAY+SZiHnz+U+FqAGBBDaDTjF2yiKa2nJSLKg@mail.gmail.com Reference Leak with type in plpgsql assignment logic]<br />
** Owner: Tom Lane<br />
<br />
* [https://www.postgresql.org/message-id/1820954.1617860500@sss.pgh.pa.us Handling of querystring inconsistent for parallel execution of SQL function bodies]<br />
** Owner: Peter Eisentraut<br />
<br />
* [https://www.postgresql.org/message-id/1884374.1617898865%40sss.pgh.pa.us Buildfarm does not test pg_stat_statements]<br />
<br />
* [https://www.postgresql.org/message-id/OSZPR01MB631017521EE6887ADC9492E8FD759@OSZPR01MB6310.jpnprd01.prod.outlook.com psql query cancellation is broken]<br />
** Owner: Peter Eisentraut<br />
** Patch available at [https://www.postgresql.org/message-id/alpine.DEB.2.22.394.2104071516470.2724014@pseudo]<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201016135230.GA23633%40alvherre.pgsql CREATE TABLE .. PARTITION OF fails to preserve tgenabled for inherited row triggers]<br />
** tgenabled lost on CREATE TABLE .. PARTITION OF, and on pg_dump, and comments on child triggers lost during pg_dump;<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer]<br />
** [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report]<br />
<br />
* [https://www.postgresql.org/message-id/a3be61d9-f44b-7fce-3dc8-d700fdfb6f48%402ndquadrant.com extract(julian) is undocumented and gives wrong result]<br />
** With reimplementation of extract to return numeric, this might be an opportune time to fix this one way or the other.<br />
<br />
* [https://www.postgresql.org/message-id/CAGRY4nwxKUS_RvXFW-ugrZBYxPFFM5kjwKT5O+0+Stuga5b4+Q@mail.gmail.com lwlock dtrace probes do unnecessary work if dtrace is compiled in but disabled]<br />
** since PG13<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/trinity-1c565d44-159f-488b-a518-caf13883134f-1611835701633%403c-app-gmx-bap78 hashagg broken by failing to spill grouping columns]<br />
** Fixed at: {{PgCommitURL|0ff865fbe50e82f17df8a9280fa01faf270b7f3f}}<br />
<br />
* [https://www.postgresql.org/message-id/CAE-ML+_EjH_fzfq1F3RJ1=XaaNG=-Jz-i3JqkNhXiLAsM3z-Ew@mail.gmail.com PITR promote bug: Checkpointer writes to older timeline]<br />
** Fixed at: {{PgCommitURL|595b9cba2ab0cdd057e02d3c23f34a8bcfd90a2d}}<br />
<br />
* [https://www.postgresql.org/message-id/YFBcRbnBiPdGZvfW%40paquier.xyz Permission failures with WAL files in 13~ on Windows]<br />
** Fixed at: {{PgCommitURL|78c24e97dd189f62187a99ef84016d0eb35a7978}}<br />
<br />
* [https://www.postgresql.org/message-id/CANiYTQsU7yMFpQYnv=BrcRVqK_3U3mtAzAsJCaqtzsDHfsUbdQ@mail.gmail.com CLOBBER_CACHE Server crashed with segfault 11 while executing clusterdb]<br />
** Fixed at: {{PgCommitURL|9d523119fd38fd205cb9c8ea8e7cceeb54355818}}<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
* [https://www.postgresql.org/message-id/20210216064214.GI28165%40telsasoft.com progress reporting for partitioned REINDEX]<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 14beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gCXDSmFs2c%3DR%2BVGgn7FiYcLCsEFEuDNNLGfoha%3DpBE_g%40mail.gmail.com Assertion fail with window function and nested partitioned tables]<br />
** [https://www.postgresql.org/message-id/87sg8tqhsl.fsf@aurora.ydns.eu Older report]<br />
** Fixed at: {{PgCommitURL|fb2d645dd53ff571572d830e830fc8c368063802}}<br />
<br />
* [https://www.postgresql.org/message-id/1df88660-6f08-cc6e-b7e2-f85296a2bdab@oss.nttdata.com Atomic initialization of waitStart done at backend startup]<br />
** Fixed at: {{PgCommitURL|f05ed5a5cfa55878baa77a1e39d68cb09793b477}}<br />
<br />
* [https://www.postgresql.org/message-id/20210117215940.GE8560%40telsasoft.com pg_collation_actual_version() ERROR: cache lookup failed for collation 123]<br />
** Fixed at: {{PgCommitURL|0fb0a0503bfc125764c8dba4f515058145dc7f8b}}<br />
<br />
* [https://www.postgresql.org/message-id/fd3ba610085f1ff54623478cf2f7adf5af193cbb.camel@vmware.com cryptohash: missing locking functions for OpenSSL <= 1.0.2?]<br />
** Fixed at: {{PgCommitURL|2c0cefcd18161549e9e8b103f46c0f65fca84d99}}<br />
<br />
* [https://www.postgresql.org/message-id/CAHut%2BPuPGGASnh2Dy37VYODKULVQo-5oE%3DShc6gwtRizDt%3D%3DcA%40mail.gmail.com pg_subscription - substream column?]<br />
** Fixed at: {{PgCommitURL|7efeb214ad832fa96ea950d0906b1d2b96316d15}}<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gcs0zGOp6JXU2mMVdthYhuQpFk%3DS3V8DOKT%3DLZC1L36Q%40mail.gmail.com TOAST compression method of index columns]<br />
** Fixed at: {{PgCommitURL|5db1fd7823a1a12e2bdad98abc8e102fd71ffbda}}<br />
<br />
* [https://www.postgresql.org/message-id/20210402235337.GA4082@ahch-to Crash with encoding conversion functions]<br />
** Fixed at: {{PgCommitURL|c4c393b3ec83ceb4b4d7f37cdd5302126377d069}}<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Feature Freeze: April 7, 2021 ('''Last Day to Commit Features''')<br />
* Beta 1: <br />
* Beta 2: <br />
* Beta 3: <br />
* RC 1: <br />
* GA: <br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_14_Open_Items&diff=35881PostgreSQL 14 Open Items2021-04-08T17:07:28Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please place new open items at the end of the list.<br />
<br />
* [https://www.postgresql.org/message-id/CAD21AoA%3D%3Df2VSw3c-Cp_y%3DWLKHMKc1D6s7g3YWsCOvgaYPpJcg%40mail.gmail.com Performance degradation of REFRESH MATERIALIZED VIEW]<br />
<br />
* On Windows, collation version lookup (sometimes?) fails for names like "English_United States.1252", but works for names like "en-US" (BCP47 language tags). Unfortunately the default value chosen by initdb is in the former style (users would have to use something like initdb --lc-collate=en-US for the version reporting to work).<br />
** {{PgCommitURL|9f12a3b95dd56c897f1aa3d756d8fb419e84a187}} was committed to tolerate the failure so at least we don't raise an error, but unfortunately we have no version information<br />
** We should either figure out how to improve this situation (translate the names, but how? and when? ask the OS for a default in the BCP47 style? how?), or document the limitation<br />
<br />
* [https://www.postgresql.org/message-id/20210319185247.ldebgpdaxsowiflw@alap3.anarazel.de Replication slot stats misgivings]<br />
<br />
* [https://postgr.es/m/CA+TgmobwnGawnxufvqLCrcTy4HRhMepFiXQLY8YpVD+PTuwagA@mail.gmail.com Update TOAST documentation for LZ4 compression]<br />
** Owner: Robert Haas<br />
<br />
* [https://www.postgresql.org/message-id/YFnWBYinNf1s0Y6v@msg.df7cb.de pg_regress and tablespace removal]<br />
** Owner: Michael Paquier<br />
<br />
* [https://www.postgresql.org/message-id/CAApHDvpYT10-nkSp8xXe-nbO3jmoaRyRFHbzh-RWMfAJynqgpQ@mail.gmail.com Crash with extended stats on expressions]<br />
** Owner: Tomas Vondra<br />
<br />
* [https://www.postgresql.org/message-id/CC3F964B-8FA1-4A23-9D3E-6EA00BBFF0EE@enterprisedb.com Issues in PostgresNode and older major versions with multi-install]<br />
** Owner: Andrew Dunstan<br />
<br />
* [https://www.postgresql.org/message-id/20210404220802.GA728316@rfd.leadboat.com Behavior of pg_dump --extension with schemas]<br />
** Owner: Michael Paquier<br />
<br />
* [https://www.postgresql.org/message-id/CAAV6ZkQRCVBh8qAY+SZiHnz+U+FqAGBBDaDTjF2yiKa2nJSLKg@mail.gmail.com Reference Leak with type in plpgsql assignment logic]<br />
** Owner: Tom Lane<br />
<br />
* [https://www.postgresql.org/message-id/1820954.1617860500@sss.pgh.pa.us Handling of querystring inconsistent for parallel execution of SQL function bodies]<br />
** Owner: Peter Eisentraut<br />
<br />
* [https://www.postgresql.org/message-id/1884374.1617898865%40sss.pgh.pa.us Buildfarm does not test pg_stat_statements]<br />
<br />
* [https://www.postgresql.org/message-id/OSZPR01MB631017521EE6887ADC9492E8FD759@OSZPR01MB6310.jpnprd01.prod.outlook.com psql query cancellation is broken]<br />
** Owner: Peter Eisentraut<br />
<br />
== Older bugs affecting stable branches ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br />
** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br />
** Unclear what the user impact is.<br />
** Affects all stable branches.<br />
<br />
* [https://www.postgresql.org/message-id/20201016135230.GA23633%40alvherre.pgsql CREATE TABLE .. PARTITION OF fails to preserve tgenabled for inherited row triggers]<br />
** tgenabled lost on CREATE TABLE .. PARTITION OF, and on pg_dump, and comments on child triggers lost during pg_dump;<br />
<br />
* [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br />
** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer]<br />
** [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report]<br />
<br />
* [https://www.postgresql.org/message-id/a3be61d9-f44b-7fce-3dc8-d700fdfb6f48%402ndquadrant.com extract(julian) is undocumented and gives wrong result]<br />
** With reimplementation of extract to return numeric, this might be an opportune time to fix this one way or the other.<br />
<br />
* [https://www.postgresql.org/message-id/CAGRY4nwxKUS_RvXFW-ugrZBYxPFFM5kjwKT5O+0+Stuga5b4+Q@mail.gmail.com lwlock dtrace probes do unnecessary work if dtrace is compiled in but disabled]<br />
** since PG13<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/flat/trinity-1c565d44-159f-488b-a518-caf13883134f-1611835701633%403c-app-gmx-bap78 hashagg broken by failing to spill grouping columns]<br />
** Fixed at: {{PgCommitURL|0ff865fbe50e82f17df8a9280fa01faf270b7f3f}}<br />
<br />
* [https://www.postgresql.org/message-id/CAE-ML+_EjH_fzfq1F3RJ1=XaaNG=-Jz-i3JqkNhXiLAsM3z-Ew@mail.gmail.com PITR promote bug: Checkpointer writes to older timeline]<br />
** Fixed at: {{PgCommitURL|595b9cba2ab0cdd057e02d3c23f34a8bcfd90a2d}}<br />
<br />
* [https://www.postgresql.org/message-id/YFBcRbnBiPdGZvfW%40paquier.xyz Permission failures with WAL files in 13~ on Windows]<br />
** Fixed at: {{PgCommitURL|78c24e97dd189f62187a99ef84016d0eb35a7978}}<br />
<br />
* [https://www.postgresql.org/message-id/CANiYTQsU7yMFpQYnv=BrcRVqK_3U3mtAzAsJCaqtzsDHfsUbdQ@mail.gmail.com CLOBBER_CACHE Server crashed with segfault 11 while executing clusterdb]<br />
** Fixed at: {{PgCommitURL|9d523119fd38fd205cb9c8ea8e7cceeb54355818}}<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
* [https://www.postgresql.org/message-id/20210216064214.GI28165%40telsasoft.com progress reporting for partitioned REINDEX]<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 14beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gCXDSmFs2c%3DR%2BVGgn7FiYcLCsEFEuDNNLGfoha%3DpBE_g%40mail.gmail.com Assertion fail with window function and nested partitioned tables]<br />
** [https://www.postgresql.org/message-id/87sg8tqhsl.fsf@aurora.ydns.eu Older report]<br />
** Fixed at: {{PgCommitURL|fb2d645dd53ff571572d830e830fc8c368063802}}<br />
<br />
* [https://www.postgresql.org/message-id/1df88660-6f08-cc6e-b7e2-f85296a2bdab@oss.nttdata.com Atomic initialization of waitStart done at backend startup]<br />
** Fixed at: {{PgCommitURL|f05ed5a5cfa55878baa77a1e39d68cb09793b477}}<br />
<br />
* [https://www.postgresql.org/message-id/20210117215940.GE8560%40telsasoft.com pg_collation_actual_version() ERROR: cache lookup failed for collation 123]<br />
** Fixed at: {{PgCommitURL|0fb0a0503bfc125764c8dba4f515058145dc7f8b}}<br />
<br />
* [https://www.postgresql.org/message-id/fd3ba610085f1ff54623478cf2f7adf5af193cbb.camel@vmware.com cryptohash: missing locking functions for OpenSSL <= 1.0.2?]<br />
** Fixed at: {{PgCommitURL|2c0cefcd18161549e9e8b103f46c0f65fca84d99}}<br />
<br />
* [https://www.postgresql.org/message-id/CAHut%2BPuPGGASnh2Dy37VYODKULVQo-5oE%3DShc6gwtRizDt%3D%3DcA%40mail.gmail.com pg_subscription - substream column?]<br />
** Fixed at: {{PgCommitURL|7efeb214ad832fa96ea950d0906b1d2b96316d15}}<br />
<br />
* [https://www.postgresql.org/message-id/CAJKUy5gcs0zGOp6JXU2mMVdthYhuQpFk%3DS3V8DOKT%3DLZC1L36Q%40mail.gmail.com TOAST compression method of index columns]<br />
** Fixed at: {{PgCommitURL|5db1fd7823a1a12e2bdad98abc8e102fd71ffbda}}<br />
<br />
* [https://www.postgresql.org/message-id/20210402235337.GA4082@ahch-to Crash with encoding conversion functions]<br />
** Fixed at: {{PgCommitURL|c4c393b3ec83ceb4b4d7f37cdd5302126377d069}}<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
<br />
* Feature Freeze: April 7, 2021 ('''Last Day to Commit Features''')<br />
* Beta 1: <br />
* Beta 2: <br />
* Beta 3: <br />
* RC 1: <br />
* GA: <br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Monitoring&diff=35666Monitoring2021-02-11T12:08:16Z<p>Rjuju: </p>
<hr />
<div>{{Languages}}<br />
<br />
== PostgreSQL builtin & contrib ==<br />
<br />
=== Statistics collector ===<br />
<br />
PostgreSQL collects lots of data on its own and offers it via the <tt>pg_stat(io)_</tt> system views<br />
<br />
* Official documentation on the [http://www.postgresql.org/docs/current/static/monitoring-stats.html Statistics Collector]<br />
* [http://www.varlena.com/GeneralBits/107.php Interpreting pg_stat Views]<br />
<br />
=== contrib extensions ===<br />
<br />
The following extensions offer access to Postgres internals which may be of interest or collect additional information. Most of them are shipped with Postgres (the <tt>-contrib</tt> packages may need to be installed) and can be activated via the [http://www.postgresql.org/docs/current/static/sql-createextension.html extension interface].<br />
<br />
==== pg_stat_statements ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstatstatements.html pg_stat_statements]</tt> tracks all queries that are executed on the server and records average runtime per query "class" among other parameters.<br />
<br />
==== pg_stat_plans ====<br />
<br />
<tt>[https://github.com/2ndQuadrant/pg_stat_plans pg_stat_plans]</tt> extends on pg_stat_statements and records query plans for all executed quries. This is very helpful when you're experiencing performance regressions due to inefficient query plans due to changed parameters or table sizes.<br />
<br />
==== pgstattuple ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstattuple.html pgstattuple]</tt> can generate statistics for tables and indexes, showing how much space in each table & index is consumed by live tuples, deleted tuples as well as how much unused space is available in each relation.<br />
<br />
==== pg_buffercache ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgbuffercache.html pg_buffercache]</tt> gives you introspection into Postgres' [http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-SHARED-BUFFERS shared buffers], showing how many pages of which relations are currently held in the cache.<br />
<br />
== External projects ==<br />
<br />
=== CLI tools ===<br />
<br />
==== pg_view ====<br />
<br />
<tt>[https://github.com/zalando/pg_view/ pg_view]</tt> is a Python-based tool to quickly get information about running databases and resources used by them as well as correlate running queries and why they might be slow.<br />
<br />
==== pg_activity ====<br />
<br />
<tt>[https://github.com/julmon/pg_activity pg_activity]</tt> is a htop like application for PostgreSQL server activity monitoring, written in Python.<br />
<br />
==== pgmetrics ====<br />
<br />
<tt>[https://pgmetrics.io/ pgmetrics]</tt> collects a lot of information and statistics from a running PostgreSQL server and displays it in easy-to-read text format or export it as JSON for scripting.<br />
<br />
==== pgstats ====<br />
<br />
<tt>[https://github.com/gleu/pgstats pgstats]</tt> is a command line tool written in C which can sample various PostgreSQL informations. It also provides a tool to generate CSV files to graph the pgstats metrics.<br />
<br />
==== pgcenter ====<br />
<br />
<tt>[https://pgcenter.org pgcenter]</tt> is an admin tool for working with PostgreSQL stats, written in Golang. It provides top like viewer with a few admin functions, tool for recording stats into files and building reports.<br />
<br />
=== Checkers ===<br />
<br />
==== check_pgactivity ====<br />
<br />
<tt>[https://github.com/OPMDG/check_pgactivity check_pgactivity]</tt> is designed to monitor PostgreSQL clusters from any Nagios like software. It offers many options to measure and monitor useful performance metrics.<br />
<br />
==== check_postgres ====<br />
<br />
<tt>[http://bucardo.org/wiki/Check_postgres check_postgres]</tt> is a command line tool which is designed to be run from software like Icinga, MRTG or as a standalone tool. It can monitor many aspects of the database and trigger warnings when thresholds are violated.<br />
<br />
=== Interfaces & collectors ===<br />
<br />
These tools either offer an interface to PostgreSQL monitoring-relevant data or can aggregate and prepare them for other systems.<br />
<br />
==== pgsnmpd ====<br />
<br />
<tt>[http://pgsnmpd.projects.postgresql.org/ pgsnmpd]</tt> can run as a standalone SNMP server and implements the [http://www.faqs.org/rfcs/rfc1697.html RFC 1697 MIB] which is generic RDBMS [http://en.wikipedia.org/wiki/Management_information_base MIB]<br />
<br />
This is useful for network management systems which are limited to SNMP protocol.<br />
<br />
==== pganalyze/collector ====<br />
<br />
<tt>[https://github.com/pganalyze/collector pganalyze/collector]</tt> is a tool which collects <tt>pg_stat_plans</tt> query information as well as various performance-relevant database parameters and converts them into a JSON structure for easy ingestion in other systems.<br />
<br />
=== Generic monitoring solutions with plugins ===<br />
<br />
==== Zabbix ====<br />
<br />
[http://pg-monz.github.io/pg_monz/index-en.html pg_monz] is a [http://www.zabbix.com/ Zabbix] monitoring template for PostgreSQL.<br />
<br />
[http://cavaliercoder.github.io/libzbxpgsql/ libzbxpgsql] is a [http://www.zabbix.com/ Zabbix] monitoring template and native agent module for PostgreSQL.<br />
<br />
==== Munin ====<br />
<br />
PostgreSQL Plugins developed in Perl are included in the Core [http://munin-monitoring.org/ Munin] Distribution. The following plugins are included by default: <tt>postgres_bgwriter, postgres_locks_, postgres_tuples_, postgres_cache_, postgres_querylength_, postgres_users, postgres_checkpoints, postgres_scans_, postgres_xlog, postgres_connections_, postgres_size_, postgres_connections_db, postgres_transactions_</tt><br />
<br />
[http://aouyar.github.com/PyMunin/ PyMunin] includes a Multigraph Munin Plugin written in Python that implements the following graphs: <tt>pg_connections, pg_diskspace, pg_blockreads, pg_xact, pg_tup_read, pg_tup_write, pg_blockreads_detail, pg_xact_commit_detail, pg_xact_rollback_detail, pg_tup_return_detail, pg_tup_fetch_detail, pg_tup_delete_detail, pg_tup_update_detail, pg_tup_insert_detail</tt><br />
<br />
<br />
==== NewRelic ====<br />
<br />
[http://newrelic.com/ NewRelic] is a proprietary SaaS application monitoring solution which offers a [https://newrelic.com/plugins/enterprisedb-corporation/30 PostgreSQL plugin] maintained by EnterpriseDB.<br />
<br />
====Datadog====<br />
[http://www.datadoghq.com/ Datadog] is a proprietary saas that collects postgres metrics on connections, transactions, row crud operations, locks, temp files, bgwriter, index usage, replication status, memory, disk, cpu and lets you visualize and alert on those metrics alongside your other system and application metrics.<br />
<br />
==== Circonus ====<br />
[https://www.circonus.com Circonus] is a general purpose monitoring, analytic and alerting saas that has predefined queries for postgres to monitor some of the common metrics and checks like connections, transactions, WALs, vacuum and table stats. More information can be found [https://login.circonus.com/resources/docs/user/Data/CheckTypes/PostgreSQL.html here]<br />
<br />
====ClusterControl by Severalnines====<br />
<br />
[https://severalnines.com/product/clustercontrol/for_postgresql ClusterControl] is an all-inclusive open source database management system that allows you to deploy, monitor, manage and scale your database environments. ClusterControl provides the basic functionality you need to get PostgreSQL up-and-running using the deployment wizard. It offers Advanced Performance Monitoring - ClusterControl monitors queries and detects anomalies with built-in alerts. Deployment and monitoring are free, with management features as part of a paid version.<br />
<br />
==== Cacti ====<br />
<br />
There has been work done on building a Postgres template for [http://www.cacti.net/ Cacti], Details can be found at the [[Cacti]] page.<br />
<br />
====Okmeter====<br />
<br />
[http://okmeter.io/pg Okmeter.io] is a proprietary SaaS that collects a whole lot of PostgreSQL status and operational data: over 50 types of metrics on connections, transactions, table CRUD operations, locks, bgwriter, index usage and ops, replication, autovacuum. Also, query timings, disk and CPU usage by queries from pg_stat_statements, and system metrics — CPU, memory, fd and disk usage per process, socket connections per port and tcp status. Collecting the data requires minimal to no configuration, there's pre-built [https://okmeter.io/example/hosts/db-postgresql1/Postgresql chart dashboards], detailed query reports and pre-set alerts, that will notify you if something's wrong with you DB. [https://okmeter.io/pg?lang=en More information here] and [https://okmeter.io/i/integrations/postgresql-monitoring?lang=en detailed info of what's collected - here.]<br />
<br />
==== Sematext ====<br />
<br />
[https://sematext.com/cloud/ Sematext Cloud] is a monitoring SaaS that collects PostgreSQL metrics such as connections, transactions, row CRUD and index statistics, WAL archiver, brwriter and more. Complete list of metrics is [https://sematext.com/docs/integration/postgresql/#metrics here]. Metrics can be correlated with data from logs (e.g. statement time), via the [https://sematext.com/docs/integration/postgresql-logs/ Sematext PostgreSQL Logs integration]. An on-permises variant of Sematext Cloud is available as [https://sematext.com/enterprise/ Sematext Enterprise].<br />
<br />
=== Postgres-centric monitoring solutions ===<br />
<br />
==== EnterpriseDB Postgres Enterprise Manager ====<br />
<br />
[http://www.enterprisedb.com/products-services-training/products/postgres-enterprise-manager Postgres Enterprise Manager] monitors, alerts, manages and tunes local and remote large scale Postgres deployments from a single graphical console. Out of the box features include: server auto-discovery, point and click management of database objects, 225+ pre-configured database alerts by SMTP/SNMP, custom alerts, global At-a-Glance monitoring dashboards, Performance monitoring dashboards, custom dashboards, an Audit Manager, Postgres Expert best practice configuration recommendations, a Log Manager, a Log Analyzer Expert, a SQL Profiler, and Index Advisor.<br />
<br />
==== pganalyze ====<br />
<br />
[https://pganalyze.com/ pganalyze] is a proprietary SaaS offering which focuses on performance monitoring and automated tuning suggestions.<br />
<br />
==== pgwatch2 ====<br />
<br />
[https://github.com/cybertec-postgresql/pgwatch2 pgwatch2] is a self-contained, easy to install and highly configurable PostgreSQL monitoring tool. It is dockerized, features a dashboard and can send alerts. No extensions or superuser privileges required!<br />
<br />
==== pg_statsinfo & pg_stats_reporter ====<br />
<br />
[http://pgstatsinfo.sourceforge.net/ pg_statsinfo] is a Postgres extension that collects lots of performance-relevant information inside the Postgres server which then can be aggregated by pg_stats_reporter instances which provide a web interface to the collected data. Both are FOSS software maintained by NTT.<br />
<br />
==== PGObserver ====<br />
<br />
[http://zalando.github.io/PGObserver/ PGObserver] is a Python & Java-based Postgres monitoring solution developed by Zalando. It was developed with a focus on stored procedure performance but extended well beyond that.<br />
<br />
==== pgCluu ====<br />
<br />
[http://pgcluu.darold.net/ pgCluu] is a Perl-based monitoring solution which uses psql and [http://en.wikipedia.org/wiki/Sar_(Unix) sar] to collect information about Postgres servers and render comprehensive performance stats.<br />
<br />
==== PoWA ====<br />
<br />
[https://powa.readthedocs.io/ PoWA] is a PostgreSQL Workload Analyzer that gathers performance stats and provides real-time charts and graphs to help monitor and tune your PostgreSQL servers. It relies on extensions such as pg_stat_statements, pg_qualstats, pg_stat_kcache, pg_track_settings and HypoPG, and can help you optimize you database easily. It's entirely open-source and free.<br />
<br />
An online demo is available at [https://demo-powa.anayrat.info/ demo-powa.anayrat.info]. Just click "login" to start using it.<br />
<br />
==== OPM: Open PostgreSQL Monitoring ====<br />
<br />
[http://opm.io Open PostgreSQL Monitoring (OPM)] is a free software suite designed to help you manage your PostgreSQL servers. It's a flexible tool that will follow the activity of each instance. It can gather stats, display dashboards and send warnings when something goes wrong. The long-term goal of the project is to provide similar features to those of Oracle Grid Control or SQL Server Management Studio.<br />
<br />
==== PASH-Viewer: PostgreSQL Active Session History Viewer ====<br />
<br />
[https://github.com/dbacvetkov/PASH-Viewer PASH-Viewer] is a free open-source software which provides graphical view of active session history and help you to answer questions like "What wait events were taking most time?", "Which sessions were taking most time?", "Which queries were taking most time and what were they doing?". It also supports Active Session History extension by pgsentinel.<br />
<br />
==== Datasentinel ====<br />
<br />
[https://www.datasentinel.io/ Datasentinel] is a proprietary monitoring and troubleshooting solution (SaaS or On-Premises ) helping you to quickly identify slowdowns thanks to its many features (sessions workload, complete activity statistics of sqls, databases, instances, and much more...)<br />
<br />
[[Category:Monitoring|!]]<br />
[[Category:Performance]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_13_Open_Items&diff=34727PostgreSQL 13 Open Items2020-03-29T08:01:13Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please add new open items to the bottom of the list.<br />
<br />
* [https://www.postgresql.org/message-id/flat/CA%2BhUKGJUw08dPs_3EUcdO6M90GnjofPYrWp4YSLaBkgYwS-AqA%40mail.gmail.com Should we rename effective_io_concurrency?] It now has a different meaning and you might want to turn it up higher, though the default behaviour hasn't changed.<br />
** Commit: {{PgCommitURL|b09ff53667}}<br />
** Owner: Thomas Munro<br />
* [https://www.postgresql.org/message-id/flat/20200315234833.GA31110%40alvherre.pgsql#c50e981ed9dd24101c0ec054d5511d7f control max length of parameter values logged]<br />
** Commit: [https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=ba79cb5dc ba79cb5dc]<br />
<br />
== Older Bugs ==<br />
<br />
=== Live issues ===<br />
* [https://www.postgresql.org/message-id/flat/20200323165059.GA24950%40alvherre.pgsql hash node details apparently accessing pfreed allocation]<br />
** Commit: {{PgCommitURL|3fc6e2d7f}}<br />
* [https://www.postgresql.org/message-id/20200328151721.GB12854%40nol pg_stat_statements doesn't report buffer consumption from parallel utility workers]<br />
** Commit: {{PgCommitURL|9da0cc35284bdbe8d442d732963303ff0e0a40bc}} (parallel btree index build) - {{PgCommitURL|40d964ec997f64227bc0ff5e058dc4a5770a70a9}} (parallel vacuum)<br />
<br />
=== Fixed issues ===<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 13rc1 ===<br />
<br />
=== resolved before 13beta2 ===<br />
<br />
=== resolved before 13beta1 ===<br />
* [https://www.postgresql.org/message-id/flat/20200310190142.GB29065%40telsasoft.com#78420835db672ec62b83e00789efb367 "backend type in log_line_prefix": Update file_fdw]<br />
** Fixed at {{PgCommitURL|0830d21f5b01064837dc8bd910ab31a5b7a1101a}}<br />
<br />
* [https://www.postgresql.org/message-id/753391579708726@iva3-77ae5995f07f.qloud-c.yandex.net Rework handling of wal_receiver_create_temp_slot to fit with WAL receiver architecture]<br />
** Fixed at {{PgCommitURL|092c6936de49e}}<br />
<br />
* [https://www.postgresql.org/message-id/flat/CAMbWs4_maqdBnRR4x01pDpoV-CiQ%2BRvMQaPm4JoTPbA%3DmZmhMw%40mail.gmail.com Negative cost is seen for plan node: Hash agg spill to disk]<br />
** Fixed at: {{PgCommitURL|7351bfeda33b60b69c15791c7eb77a127546df26}}<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
* feature freeze: XXX<br />
* beta1: XXX<br />
* beta2: XXX<br />
* rc1: XXX<br />
* GA: XXX<br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_13_Open_Items&diff=34724PostgreSQL 13 Open Items2020-03-28T15:19:07Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please add new open items to the bottom of the list.<br />
<br />
* [https://www.postgresql.org/message-id/flat/CA%2BhUKGJUw08dPs_3EUcdO6M90GnjofPYrWp4YSLaBkgYwS-AqA%40mail.gmail.com Should we rename effective_io_concurrency?] It now has a different meaning and you might want to turn it up higher, though the default behaviour hasn't changed.<br />
** Commit: {{PgCommitURL|b09ff53667}}<br />
** Owner: Thomas Munro<br />
* [https://www.postgresql.org/message-id/flat/20200315234833.GA31110%40alvherre.pgsql#c50e981ed9dd24101c0ec054d5511d7f control max length of parameter values logged]<br />
** Commit: [https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=ba79cb5dc ba79cb5dc]<br />
* [https://www.postgresql.org/message-id/flat/CAMbWs4_maqdBnRR4x01pDpoV-CiQ%2BRvMQaPm4JoTPbA%3DmZmhMw%40mail.gmail.com Negative cost is seen for plan node: Hash agg spill to disk]<br />
** Commit: {{PgCommitURL|1f39bce021540fde00990af55b4432c55ef4b3c7}}<br />
** Owner: Jeff Davis<br />
* [https://www.postgresql.org/message-id/20200328151721.GB12854%40nol pg_stat_statements doesn't report buffer consumption from parallel utility workers]<br />
** Commit: {{PgCommitURL|9da0cc35284bdbe8d442d732963303ff0e0a40bc}} (parallel btree index build) - {{PgCommitURL|40d964ec997f64227bc0ff5e058dc4a5770a70a9}} (parallel vacuum)<br />
<br />
<br />
== Older Bugs ==<br />
<br />
=== Live issues ===<br />
* [https://www.postgresql.org/message-id/flat/20200323165059.GA24950%40alvherre.pgsql hash node details apparently accessing pfreed allocation]<br />
** Commit: {{PgCommitURL|3fc6e2d7f}}<br />
<br />
=== Fixed issues ===<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 13rc1 ===<br />
<br />
=== resolved before 13beta2 ===<br />
<br />
=== resolved before 13beta1 ===<br />
* [https://www.postgresql.org/message-id/flat/20200310190142.GB29065%40telsasoft.com#78420835db672ec62b83e00789efb367 "backend type in log_line_prefix": Update file_fdw]<br />
** Fixed at {{PgCommitURL|0830d21f5b01064837dc8bd910ab31a5b7a1101a}}<br />
<br />
* [https://www.postgresql.org/message-id/753391579708726@iva3-77ae5995f07f.qloud-c.yandex.net Rework handling of wal_receiver_create_temp_slot to fit with WAL receiver architecture]<br />
** Fixed at {{PgCommitURL|092c6936de49e}}<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
* feature freeze: XXX<br />
* beta1: XXX<br />
* beta2: XXX<br />
* rc1: XXX<br />
* GA: XXX<br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_13_Open_Items&diff=34723PostgreSQL 13 Open Items2020-03-28T14:47:42Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please add new open items to the bottom of the list.<br />
<br />
* [https://www.postgresql.org/message-id/flat/CA%2BhUKGJUw08dPs_3EUcdO6M90GnjofPYrWp4YSLaBkgYwS-AqA%40mail.gmail.com Should we rename effective_io_concurrency?] It now has a different meaning and you might want to turn it up higher, though the default behaviour hasn't changed.<br />
** Commit: {{PgCommitURL|b09ff53667}}<br />
** Owner: Thomas Munro<br />
* [https://www.postgresql.org/message-id/flat/20200315234833.GA31110%40alvherre.pgsql#c50e981ed9dd24101c0ec054d5511d7f control max length of parameter values logged]<br />
** Commit: [https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=ba79cb5dc ba79cb5dc]<br />
* [https://www.postgresql.org/message-id/flat/CAMbWs4_maqdBnRR4x01pDpoV-CiQ%2BRvMQaPm4JoTPbA%3DmZmhMw%40mail.gmail.com Negative cost is seen for plan node: Hash agg spill to disk]<br />
** Commit: {{PgCommitURL|1f39bce021540fde00990af55b4432c55ef4b3c7}}<br />
** Owner: Jeff Davis<br />
* [https://www.postgresql.org/message-id/20200328133827.GA12854@nol pg_stat_statements doesn't report buffer consumption from parallel utility workers]<br />
** Commit: {{PgCommitURL|9da0cc35284bdbe8d442d732963303ff0e0a40bc}} (parallel btree index build) - {{PgCommitURL|40d964ec997f64227bc0ff5e058dc4a5770a70a9}} (parallel vacuum)<br />
<br />
<br />
== Older Bugs ==<br />
<br />
=== Live issues ===<br />
* [https://www.postgresql.org/message-id/flat/20200323165059.GA24950%40alvherre.pgsql hash node details apparently accessing pfreed allocation]<br />
** Commit: {{PgCommitURL|3fc6e2d7f}}<br />
<br />
=== Fixed issues ===<br />
<br />
=== Nothing to do ===<br />
<br />
== Non-bugs ==<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 13rc1 ===<br />
<br />
=== resolved before 13beta2 ===<br />
<br />
=== resolved before 13beta1 ===<br />
* [https://www.postgresql.org/message-id/flat/20200310190142.GB29065%40telsasoft.com#78420835db672ec62b83e00789efb367 "backend type in log_line_prefix": Update file_fdw]<br />
** Fixed at {{PgCommitURL|0830d21f5b01064837dc8bd910ab31a5b7a1101a}}<br />
<br />
* [https://www.postgresql.org/message-id/753391579708726@iva3-77ae5995f07f.qloud-c.yandex.net Rework handling of wal_receiver_create_temp_slot to fit with WAL receiver architecture]<br />
** Fixed at {{PgCommitURL|092c6936de49e}}<br />
<br />
== Won't Fix ==<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
* feature freeze: XXX<br />
* beta1: XXX<br />
* beta2: XXX<br />
* rc1: XXX<br />
* GA: XXX<br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2020_Developer_Meeting&diff=34401FOSDEM/PGDay 2020 Developer Meeting2019-11-24T17:06:56Z<p>Rjuju: </p>
<hr />
<div>A meeting for PostgreSQL developers will be held in conjunction with the PostgreSQL Europe FOSDEM Devroom<br />
and FOSDEM/PGDay 2020 events in Brussels, Belgium. The meeting is planned for January 30. In order to keep the meeting focused, productive, and with a high level of interaction, the meeting is invite-only. Due to the large developer community, we cannot invite every active contributor. If you feel that you, or someone else, should've been invited then please contact [mailto:daniel@yesql.se Daniel Gustafsson].<br />
<br />
This is a Community event, organized and financed by PostgreSQL Europe.<br />
<br />
== Time and Location ==<br />
The meeting will be held at the [https://www3.hilton.com/en/hotels/brussels-capital-reg/hilton-brussels-grand-place-BRUGRHI/index.html Hilton Brussels Grand Place Hotel], which is the venue where [https://2020.fosdempgday.org/ FOSDEM PGDay 2020] is arranged the day after on January 31. The meeting starts at 9AM and is scheduled to end at 5PM. Coffee, tea and snacks will be provided during the day, as well as lunch.<br />
<br />
== Attendees ==<br />
The following hackers have RSVP'ed to the meeting and will be attending:<br />
* Daniel Gustafsson<br />
* Joe Conway<br />
* Alexander Korotkov<br />
* Julien Rouhaud<br />
<br />
== Suggested Topics ==<br />
Please add topics for discussion to the list:<br />
* CI build feedback/information directly in CF app (Daniel)<br />
* Commitfest triage<br />
<br />
== Agenda ==<br />
<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Item<br />
!Presenter<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|09:00 - 09:10<br />
|Welcome and introductions<br />
|Daniel<br />
<br />
|- <br />
|09:10 - 09:20<br />
|TBD<br />
|All<br />
<br />
|- <br />
|09:20 - 09:40<br />
|TBD<br />
|All<br />
<br />
|- <br />
|09:40 - 10:00<br />
|TBD<br />
|All<br />
<br />
|- <br />
|10:00 - 10:20<br />
|TBD<br />
|All<br />
<br />
|- <br />
|10:20 - 10:30<br />
|TBD<br />
|All<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|10:30 - 11:00<br />
|Coffee break<br />
|<br />
<br />
|- <br />
|11:00 - 11:15<br />
|TBD<br />
|All<br />
<br />
|- <br />
|11:15 - 12:30<br />
|TBD<br />
|All<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|12:30 - 13:30<br />
|Lunch<br />
|<br />
<br />
|- <br />
|13:30 - 15:00<br />
|Commitfest Triage<br />
|All<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|15:00 - 15:30<br />
|Tea break<br />
|<br />
<br />
|- <br />
|15:30 - 16:45<br />
|Commitfest Triage<br />
|All<br />
<br />
<br />
|- <br />
|16:45 - 17:00<br />
|Any other business, plans for next year<br />
|Daniel<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|17:00<br />
|Finish<br />
|<br />
|}</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Monitoring&diff=33681Monitoring2019-07-21T18:32:36Z<p>Rjuju: </p>
<hr />
<div>{{Languages}}<br />
<br />
== PostgreSQL builtin & contrib ==<br />
<br />
=== Statistics collector ===<br />
<br />
PostgreSQL collects lots of data on its own and offers it via the <tt>pg_stat(io)_</tt> system views<br />
<br />
* Official documentation on the [http://www.postgresql.org/docs/current/static/monitoring-stats.html Statistics Collector]<br />
* [http://www.varlena.com/GeneralBits/107.php Interpreting pg_stat Views]<br />
<br />
=== contrib extensions ===<br />
<br />
The following extensions offer access to Postgres internals which may be of interest or collect additional information. Most of them are shipped with Postgres (the <tt>-contrib</tt> packages may need to be installed) and can be activated via the [http://www.postgresql.org/docs/current/static/sql-createextension.html extension interface].<br />
<br />
==== pg_stat_statements ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstatstatements.html pg_stat_statements]</tt> tracks all queries that are executed on the server and records average runtime per query "class" among other parameters.<br />
<br />
==== pg_stat_plans ====<br />
<br />
<tt>[https://github.com/2ndQuadrant/pg_stat_plans pg_stat_plans]</tt> extends on pg_stat_statements and records query plans for all executed quries. This is very helpful when you're experiencing performance regressions due to inefficient query plans due to changed parameters or table sizes.<br />
<br />
==== pgstattuple ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstattuple.html pgstattuple]</tt> can generate statistics for tables and indexes, showing how much space in each table & index is consumed by live tuples, deleted tuples as well as how much unused space is available in each relation.<br />
<br />
==== pg_buffercache ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgbuffercache.html pg_buffercache]</tt> gives you introspection into Postgres' [http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-SHARED-BUFFERS shared buffers], showing how many pages of which relations are currently held in the cache.<br />
<br />
== External projects ==<br />
<br />
=== CLI tools ===<br />
<br />
==== pg_view ====<br />
<br />
<tt>[https://github.com/zalando/pg_view/ pg_view]</tt> is a Python-based tool to quickly get information about running databases and resources used by them as well as correlate running queries and why they might be slow.<br />
<br />
==== pg_activity ====<br />
<br />
<tt>[https://github.com/julmon/pg_activity pg_activity]</tt> is a htop like application for PostgreSQL server activity monitoring, written in Python.<br />
<br />
==== pgmetrics ====<br />
<br />
<tt>[https://pgmetrics.io/ pgmetrics]</tt> collects a lot of information and statistics from a running PostgreSQL server and displays it in easy-to-read text format or export it as JSON for scripting.<br />
<br />
==== pgstats ====<br />
<br />
<tt>[https://github.com/gleu/pgstats pgstats]</tt> is a command line tool written in C which can sample various PostgreSQL informations. It also provides a tool to generate CSV files to graph the pgstats metrics.<br />
<br />
==== pgcenter ====<br />
<br />
<tt>[https://pgcenter.org pgcenter]</tt> is an admin tool for working with PostgreSQL stats, written in Golang. It provides top like viewer with a few admin functions, tool for recording stats into files and building reports.<br />
<br />
=== Checkers ===<br />
<br />
==== check_pgactivity ====<br />
<br />
<tt>[https://github.com/OPMDG/check_pgactivity check_pgactivity]</tt> is designed to monitor PostgreSQL clusters from any Nagios like software. It offers many options to measure and monitor useful performance metrics.<br />
<br />
==== check_postgres ====<br />
<br />
<tt>[http://bucardo.org/wiki/Check_postgres check_postgres]</tt> is a command line tool which is designed to be run from software like Icinga, MRTG or as a standalone tool. It can monitor many aspects of the database and trigger warnings when thresholds are violated.<br />
<br />
=== Interfaces & collectors ===<br />
<br />
These tools either offer an interface to PostgreSQL monitoring-relevant data or can aggregate and prepare them for other systems.<br />
<br />
==== pgsnmpd ====<br />
<br />
<tt>[http://pgsnmpd.projects.postgresql.org/ pgsnmpd]</tt> can run as a standalone SNMP server and implements the [http://www.faqs.org/rfcs/rfc1697.html RFC 1697 MIB] which is generic RDBMS [http://en.wikipedia.org/wiki/Management_information_base MIB]<br />
<br />
This is useful for network management systems which are limited to SNMP protocol.<br />
<br />
==== pganalyze/collector ====<br />
<br />
<tt>[https://github.com/pganalyze/collector pganalyze/collector]</tt> is a tool which collects <tt>pg_stat_plans</tt> query information as well as various performance-relevant database parameters and converts them into a JSON structure for easy ingestion in other systems.<br />
<br />
=== Generic monitoring solutions with plugins ===<br />
<br />
==== Zabbix ====<br />
<br />
[http://pg-monz.github.io/pg_monz/index-en.html pg_monz] is a [http://www.zabbix.com/ Zabbix] monitoring template for PostgreSQL.<br />
<br />
[http://cavaliercoder.github.io/libzbxpgsql/ libzbxpgsql] is a [http://www.zabbix.com/ Zabbix] monitoring template and native agent module for PostgreSQL.<br />
<br />
==== Munin ====<br />
<br />
PostgreSQL Plugins developed in Perl are included in the Core [http://munin-monitoring.org/ Munin] Distribution. The following plugins are included by default: <tt>postgres_bgwriter, postgres_locks_, postgres_tuples_, postgres_cache_, postgres_querylength_, postgres_users, postgres_checkpoints, postgres_scans_, postgres_xlog, postgres_connections_, postgres_size_, postgres_connections_db, postgres_transactions_</tt><br />
<br />
[http://aouyar.github.com/PyMunin/ PyMunin] includes a Multigraph Munin Plugin written in Python that implements the following graphs: <tt>pg_connections, pg_diskspace, pg_blockreads, pg_xact, pg_tup_read, pg_tup_write, pg_blockreads_detail, pg_xact_commit_detail, pg_xact_rollback_detail, pg_tup_return_detail, pg_tup_fetch_detail, pg_tup_delete_detail, pg_tup_update_detail, pg_tup_insert_detail</tt><br />
<br />
<br />
==== NewRelic ====<br />
<br />
[http://newrelic.com/ NewRelic] is a proprietary SaaS application monitoring solution which offers a [https://newrelic.com/plugins/enterprisedb-corporation/30 PostgreSQL plugin] maintained by EnterpriseDB.<br />
<br />
====Datadog====<br />
[http://www.datadoghq.com/ Datadog] is a proprietary saas that collects postgres metrics on connections, transactions, row crud operations, locks, temp files, bgwriter, index usage, replication status, memory, disk, cpu and lets you visualize and alert on those metrics alongside your other system and application metrics.<br />
<br />
==== Circonus ====<br />
[https://www.circonus.com Circonus] is a general purpose monitoring, analytic and alerting saas that has predefined queries for postgres to monitor some of the common metrics and checks like connections, transactions, WALs, vacuum and table stats. More information can be found [https://login.circonus.com/resources/docs/user/Data/CheckTypes/PostgreSQL.html here]<br />
<br />
====ClusterControl by Severalnines====<br />
<br />
[https://severalnines.com/product/clustercontrol/for_postgresql ClusterControl] is an all-inclusive open source database management system that allows you to deploy, monitor, manage and scale your database environments. ClusterControl provides the basic functionality you need to get PostgreSQL up-and-running using the deployment wizard. It offers Advanced Performance Monitoring - ClusterControl monitors queries and detects anomalies with built-in alerts. Deployment and monitoring are free, with management features as part of a paid version.<br />
<br />
==== Cacti ====<br />
<br />
There has been work done on building a Postgres template for [http://www.cacti.net/ Cacti], Details can be found at the [[Cacti]] page.<br />
<br />
<br />
====Okmeter====<br />
<br />
[http://okmeter.io/pg Okmeter.io] is a proprietary SaaS that collects a whole lot of PostgreSQL status and operational data: over 50 types of metrics on connections, transactions, table CRUD operations, locks, bgwriter, index usage and ops, replication, autovacuum. Also, query timings, disk and CPU usage by queries from pg_stat_statements, and system metrics — CPU, memory, fd and disk usage per process, socket connections per port and tcp status. Collecting the data requires minimal to no configuration, there's pre-built [https://okmeter.io/example/hosts/db-postgresql1/Postgresql chart dashboards], detailed query reports and pre-set alerts, that will notify you if something's wrong with you DB. [https://okmeter.io/pg?lang=en More information here] and [https://okmeter.io/i/integrations/postgresql-monitoring?lang=en detailed info of what's collected - here.]<br />
<br />
<br />
=== Postgres-centric monitoring solutions ===<br />
<br />
==== EnterpriseDB Postgres Enterprise Manager ====<br />
<br />
[http://www.enterprisedb.com/products-services-training/products/postgres-enterprise-manager Postgres Enterprise Manager] monitors, alerts, manages and tunes local and remote large scale Postgres deployments from a single graphical console. Out of the box features include: server auto-discovery, point and click management of database objects, 225+ pre-configured database alerts by SMTP/SNMP, custom alerts, global At-a-Glance monitoring dashboards, Performance monitoring dashboards, custom dashboards, an Audit Manager, Postgres Expert best practice configuration recommendations, a Log Manager, a Log Analyzer Expert, a SQL Profiler, and Index Advisor.<br />
<br />
==== pganalyze ====<br />
<br />
[https://pganalyze.com/ pganalyze] is a proprietary SaaS offering which focuses on performance monitoring and automated tuning suggestions.<br />
<br />
==== pgwatch2 ====<br />
<br />
[https://github.com/cybertec-postgresql/pgwatch2 pgwatch2] is a self-contained, easy to install and highly configurable PostgreSQL monitoring tool. It is dockerized, features a dashboard and can send alerts. No extensions or superuser privileges required!<br />
<br />
==== pg_statsinfo & pg_stats_reporter ====<br />
<br />
[http://pgstatsinfo.sourceforge.net/ pg_statsinfo] is a Postgres extension that collects lots of performance-relevant information inside the Postgres server which then can be aggregated by pg_stats_reporter instances which provide a web interface to the collected data. Both are FOSS software maintained by NTT.<br />
<br />
==== PGObserver ====<br />
<br />
[http://zalando.github.io/PGObserver/ PGObserver] is a Python & Java-based Postgres monitoring solution developed by Zalando. It was developed with a focus on stored procedure performance but extended well beyond that.<br />
<br />
==== pgCluu ====<br />
<br />
[http://pgcluu.darold.net/ pgCluu] is a Perl-based monitoring solution which uses psql and [http://en.wikipedia.org/wiki/Sar_(Unix) sar] to collect information about Postgres servers and render comprehensive performance stats.<br />
<br />
==== PoWA ====<br />
<br />
[http://dalibo.github.io/powa/ PoWA] is a PostgreSQL Workload Analyzer that gathers performance stats and provides real-time charts and graphs to help monitor and tune your PostgreSQL servers. It relies on extensions such as pg_stat_statements, pg_qualstats, pg_stat_kcache, pg_track_settings and HypoPG, and can help you optimize you database easily. It's entirely open-source and free.<br />
<br />
==== OPM: Open PostgreSQL Monitoring ====<br />
<br />
[http://opm.io Open PostgreSQL Monitoring (OPM)] is a free software suite designed to help you manage your PostgreSQL servers. It's a flexible tool that will follow the activity of each instance. It can gather stats, display dashboards and send warnings when something goes wrong. The long-term goal of the project is to provide similar features to those of Oracle Grid Control or SQL Server Management Studio.<br />
<br />
==== PASH-Viewer: PostgreSQL Active Session History Viewer ====<br />
<br />
[https://github.com/dbacvetkov/PASH-Viewer PASH-Viewer] is a free open-source software which provides graphical view of active session history and help you to answer questions like "What wait events were taking most time?", "Which sessions were taking most time?", "Which queries were taking most time and what were they doing?". It also supports Active Session History extension by pgsentinel.<br />
<br />
==== Datasentinel ====<br />
<br />
[https://www.datasentinel.io/ Datasentinel] is a proprietary monitoring and troubleshooting solution (SaaS or On-Premises ) helping you to quickly identify slowdowns thanks to its many features (sessions workload, complete activity statistics of sqls, databases, instances, and much more...)<br />
<br />
[[Category:Monitoring|!]]<br />
[[Category:Performance]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_12_Open_Items&diff=33193PostgreSQL 12 Open Items2019-03-30T17:16:09Z<p>Rjuju: </p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please add new open items to the bottom of the list.<br />
<br />
<br />
* [https://www.postgresql.org/message-id/20190225074539.az6j3u464cvsoxh6@depesz.com Segfault when restoring -Fd dump on current HEAD]<br />
* [https://www.postgresql.org/message-id/flat/19465.1541636036@sss.pgh.pa.us Inadequate index locking causes Assert failure]<br />
** [https://www.postgresql.org/message-id/CAKJS1f_ZaECEN%2BGFtfrepe18ovoqsKNs8YML1BwsnZHr3fdzKQ%40mail.gmail.com Patch exists]<br />
* [https://www.postgresql.org/message-id/CAKJS1f_1c260nOt_vBJ067AZ3JXptXVRohDVMLEBmudX1YEx-A@mail.gmail.com pg_dump is broken for partition tablespaces]<br />
* [https://www.postgresql.org/message-id/21516.1552489217@sss.pgh.pa.us Debate INFO messages in ATTACH PARTITION and SET NOT NULL]<br />
* [https://www.postgresql.org/message-id/87wolmg60q.fsf@news-spur.riddles.org.uk Inlining of nested CTEs with recursive terms]<br />
* [https://www.postgresql.org/message-id/CAMkU=1x8taZfsbPkv_MsWbTtzibW_yQHXoMhF_DTtm=z2hVHDg@mail.gmail.com compiler warning in pgcrypto imath.c]<br />
* [https://www.postgresql.org/message-id/201902021315.6h6ktmmsgjmx@alvherre.pgsql remove \cset from pgbench]<br />
* [https://www.postgresql.org/message-id/DF4PR8401MB11964EDB77C860078C343BEBEE5A0@DF4PR8401MB1196.NAMPRD84.PROD.OUTLOOK.COM Indexes part of a partition tree cannot be run with REINDEX CONCURRENTLY]<br />
* [https://www.postgresql.org/message-id/flat/CABUevEzD_duH_hGyZw14o%2BkhHBw-rWSSAxbEKt5HWy2cK0Djdw%40mail.gmail.com#d8a9d175134a072dd1477c3fac96f76a Keep track of checksum failures in shared object, last failure time and pg_stat_checkums view]<br />
<br />
== Decisions to Recheck Mid-Beta ==<br />
<br />
== Older Bugs ==<br />
<br />
* [https://www.postgresql.org/message-id/9813f079-f16b-61c8-9ab7-4363cab28d80@lab.ntt.co.jp selecting from partition directly can't use constraint exclusion]<br />
** [https://commitfest.postgresql.org/23/2072 CF entry]<br />
* [https://www.postgresql.org/message-id/20181009.181536.142257785.horiguchi.kyotaro@lab.ntt.co.jp Bypass processing of wraparound autovacuums not marked as aggressive]<br />
** Problem exists since the point where aggressive vacuums have been introduced, v12 has only added extra logs to look after the impossible case of wraparound autovacuums not aggressive.<br />
* [https://www.postgresql.org/message-id/15672-b9fa7db32698269f%40postgresql.org ATPostAlterTypeCleanup causes child indexes to be recreated with wrong relfilnode]<br />
* [https://www.postgresql.org/message-id/8305.1553884377@sss.pgh.pa.us Planner's partitionwise-join code crashes under GEQO]<br />
<br />
=== Live issues ===<br />
<br />
=== Fixed issues ===<br />
<br />
== Non-bugs ==<br />
<br />
== Resolved Issues ==<br />
<br />
=== resolved before 12beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/20190322032612.GA323@alvherre.pgsql pg_partition_root crashes when using top-most parent in input]<br />
** Fixed in: {{PgCommitURL|2ab6d28d233af17987ea323e3235b2bda89b4f2e}}<br />
* [https://www.postgresql.org/message-id/CA+HiwqEGoa485g18mt9GUdF8fH4mKDgpeoc32XiW-dRUFpN5Lw@mail.gmail.com Server crash in transformPartitionRangeBounds]<br />
** Fixed in: {{PgCommitURL|cdde886d36b5a4d7ad9e1d02596f7fa1c8c129e3}}<br />
* [https://www.postgresql.org/message-id/20190326020853.GM2558@paquier.xyz Misleading errors with column references in default expressions and partition bounds]<br />
** Fixed in: {{PgCommitURL|ecfed4a12247cf4659eee6b6ea27405e35fe57f8}}<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
* feature freeze: April 7, 2019<br />
* beta1: XXX<br />
* beta2: XXX<br />
* rc1: XXX<br />
* ga: XXX<br />
<br />
[[Category:Open_Items]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2018_Developer_Unconference&diff=32892PGConf.ASIA2018 Developer Unconference2018-12-11T06:10:02Z<p>Rjuju: </p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 10 Dec, 2018 at Akihabara Convention Hall, as part of [http://www.pgconf.asia/EN/2018/ PGConf.ASIA 2018]. <br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Okinawa<br />
!Hokkaido<br />
|- style="background-color:lightgray;"<br />
|10:30-10:50<br />
|colspan="2"|Voting<br />
<br />
|-<br />
|11:00-11:50<br />
| PostgreSQL and Data Analytics<br />
| Referential Integrity Using Statement Level Triggers<br />
<br />
|- style="background-color:lightgray;"<br />
|11:50 - 13:00 <br />
|colspan="2"|Lunch<br />
<br />
|-<br />
|13:00-13:50 <br />
|Shared Buffer Manager<br />
|Hypotehtical Partitioning<br />
<br />
|-<br />
|14:00-14:50 <br />
|Partition-wise JOIN and Visibility Map<br />
|Query Monitoring<br />
<br />
|-<br />
|15:00-15:50<br />
|Shared Catalog and Relation Cache<br />
|Built In Job Scheduler<br />
<br />
|-<br />
|16:00-16:50<br />
|Per-tablespace transparent encryption<br />
|Idle Connection Scaling<br />
<br />
|- style="background-color:lightgray;"<br />
|17:00 - <br />
|colspan="2"|Beer Elsewhere<br />
<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
* TBD<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
<br />
=== Statement level referential integrity ===<br />
<br />
Oracle-style import<br />
No way to invalidate then revalidate constraints<br />
Right thing also the last thing<br />
Checks fire once per row<br />
Kevin Grittner + Thomas Munro: statement-level triggers, transition tables<br />
benchmark: 98.x% faster<br />
no code, just a PL/PgSQL script<br />
REFERENCES: INSERT/UPDATE trigger on child, DELETE trigger on parent<br />
initial pass: LEFT OUTER JOIN<br />
code is already there!<br />
transition tables don't necessarily have names<br />
enforce constraints with a single table scan! important when _appending_ to a table<br />
* CreateFKCheckTrigger<br />
* SPI_register_triger_data<br />
very seldom more statements than rows<br />
issues with the error message: just show the first row? doesn't make things worse<br />
ignore failed rows server-side; report them so app can fix them (different thing)<br />
different rejection tables for every failure?<br />
Kevin Grittner: delta relations in ALTER triggers [pgsql-hackers]<br />
* commands/tablecmds.c<br />
* RI_Initial_Check<br />
are statement-level triggers also `Trigger *`?<br />
what does the benchmark look like for one-row-at-a-time? make the row count tunable?<br />
how to upgrade? find all row-level triggers and replace them<br />
not a concern with dump/restore, but pg_upgrade needs to skip the initial check<br />
triggers have PG-reserved names; use that to detect and upgrade triggers<br />
pg_upgrade creates empty cluster; maybe no need to skip check?<br />
all that's missing is the magic to convert the trigger<br />
no catalogs are binary-linked<br />
pg_dump/pg_restore dump standard SQL; no syntax changes<br />
switch from row-level to statement-level after X rows<br />
parallel query uses a queue already<br />
suggestion: have custom non-trigger mechanism<br />
<br />
=== Query Monitoring (enhancing pg_stat_statements) ===<br />
<br />
Common complaints:<br />
<br />
* cannot see actual values used in specific queries<br />
* cannot see the query plan that was used<br />
<br />
pg_stat_statements today<br />
<br />
* dbid<br />
* userid<br />
* queryid<br />
* query<br />
<br />
Brief explanation of pg_qualstats and how it works<br />
<br />
* high overhead to deparsing<br />
* lots of quals can be slow<br />
* sampling rate of queries deparsed is < 5%<br />
<br />
Mention of pg_stat_plans and pg_hint_plans and <br />
<br />
[https://github.com/ossc-db/pg_store_plans pg_store_plans]<br />
<br />
* also uses query id<br />
* unclear LICENSE (issue opened)<br />
<br />
Issue with auto_explain, no way to link it back to pg_stat_statements<br />
<br />
Issue with how search_path can alter queries and cause pg_stat_statements to conflate unrelated but textually similar queries<br />
<br />
Discussion of query performance variance and storing historgrams, and the limitations of those histograms.<br />
<br />
Discussion of whether it would benefit us to have indicators of which queries had the most sequential scans, index scans, etc.<br />
<br />
pg_stat_kcache, discussion of its utility, ability to gather OS-level CPU usage, I/O usage for the life of the session, which with the right sampling can be interpolated to the per-query level.<br />
<br />
Discussion of tracking wait events. <br />
<br />
Should queryid be in pg_stat_activity? PostgresPro already has an extension that does this: [https://github.com/postgrespro/pg_wait_sampling pg_wait_sampling].<br />
<br />
Issue: canceled queries don't show up in pg_stat_statements.<br />
<br />
=== PostgreSQL and Data Analytics ===<br />
<br />
Karsten: Perspective from the user<br />
<br />
1) Level of parallelism the query planner is creating is not optimal - Postgres does not create enough parallelism<br><br />
2) Customers find database crashing when number of connections goes up (out of memory errors)<br />
<br />
=> You can tune the Linux to avoid crashes and instead return out of memory errors (tell Linux to not overcommit memory)<br />
<br />
Joe Conway: "What is many connections?"<br><br />
Karsten: 1000 connections<br />
<br />
Joe Conway: Thats a lot of connections, in particular for large analytics workloads<br><br />
Joe Conway: You have to shrink work_mem when you use that many connections<br />
<br />
Lukas Fittl: Is it useful to look at work_mem as a database-wide setting instead of per-connection? (there is an existing -hackers thread on this)<br><br />
Joe Conway: People have talked about resource management over the years, its one of the features that Postgres lacks that other databases have<br />
<br />
---<br />
<br />
Karsten: To the other topic, not enough parallelism - how could parallelism work better?<br><br />
Joe Conway: What are you bottlenecked on, I/O?<br><br />
Karsten: The data is cached, in memory, you have a database of 100GB of memory, and still its not working in parallel<br><br />
Karsten: Seeing this same problem when running TPC-H benchmarks<br><br />
(?): Would need to look at the specific query execution plan to understand the parallelism problems better<br><br />
(?): Memory usage is shared_buffers, per connection relation cache, etc. - not just work_mem<br />
<br />
---<br />
<br />
Joe Conway: Do we need per-user memory restrictions?<br><br />
Karsten: In our case, not so much, its typically one user running the queries<br><br />
Karsten: In our environments, queries can easily run for 30, 60 minutes, or longer<br />
<br />
Michael Paquier: There is a memory debug function in Postgres (the same that prints in OOM situations), that could be utilized<br><br />
Joe Conway: Someone needs to sit down and come up with a list of requirements on resource management<br />
<br />
Michael Paquier: One idea could be to have a per-session memory limit that is managed by each backend<br><br />
Joe Conway: Whatever you do in this area, it would have to not be overrideable per-user (from a security perspective)<br />
<br />
[ discussion around measuring whether parallel query works ]<br />
<br />
(?): There are two counters in shared memory around parallel workers launched/etc that could be exported<br />
<br />
<br />
=== Connection Pooling ===<br />
<br />
Pain Points:<br />
* low active connections / high total connections<br />
* Snapshot building / proc array lock / CSN (Commit Snapshot Number) at 10k connections<br />
* idle_in_transaction pain (out of scope)<br />
* relcache stays after query run on the idle connection<br />
* catalog cache bloat<br />
* is there a way to coax glibc to give back memory? nope.<br />
<br />
<br />
* Connection<br />
** Session<br />
*** Transaction<br />
**** Set Local<br />
<br />
Things that taint a connection<br />
* SET<br />
* PREPARE<br />
* CREATE TEMP TABLE<br />
<br />
Discussion about what about a connection can be salvaged after a user disconnects<br />
* how many active backends to retain and for how long<br />
* "no way around" the connection pooler being per-user<br />
<br />
Discussion of how to decide how long to keep which user's connections around<br />
* Master vs replica settings.<br />
* idle_backend_timeout would terminate unused connections, and commonly used users would simply be recycled enough<br />
<br />
Discussion about max reuse of connections (to deal with memory leaks)<br />
* static analyzers already rigorously check for memory leaks, so little need for capping re-use<br />
<br />
Final Thoughts / Good Next Steps:<br />
* Could this be done as an extension, and how could you avoid connection tainting.<br />
* Does socket handoff need to be done in core<br />
* Could this just be a part of pgbouncer<br />
<br />
=== Hypothetical Partitions for testing optimizations ===<br />
<br />
* like hypopg does for indexes, but with partitions<br />
<br />
challenges:<br />
* how to get a fake table name, and how this is already accomplished for index relations in hypopg<br />
* relcache does not have the ability to remove negative relcache entries<br />
* how to impose catalog entries to represent the partitions in just one session<br />
* velocity of changes to partitioning in recent versions (10,11, master) means that few people understand what current behavior is<br />
* some insert/delete hooks coming in v12, which may cut down the amount of code that needs to be duplicated<br />
* CreateFakeRelcacheEntry / FreeFakeRelcacheEntry could be used<br />
- How to select Oids for fake objects on a read replica, risk collision with new real objects from master. Some question of whether fake Oids are even necessary.<br />
<br />
It was suggested that refactoring for v12, even if incomplete, is better than no progress at all.<br />
<br />
Several user questions about hypopg and how stable it is, and how it would be used, with an eye toward including in cloud DBAAS products.<br />
<br />
Some concern was expressed that it may be too late in the development cycle to get a chance of this size into v12.<br />
<br />
=== Partition-wise Join & Visibility Map ===<br />
<br />
KaiGai Kohei (HeteroDB,Inc)<br />
<br />
First, KaiGai introduced the background why he focuses on the features.<br />
<br />
He has developed PG-Strom which is an extension of PostgreSQL to accelerate analytic queries using GPU. It also challenges to I/O intensive workloads by its SSD-to-GPU Direct SQL feature; that loads PostgreSQL's data blocks on the storage (NVME-SSD) to GPU directly using P2P DMA, then GPU runs a part of SQL workloads (WHERE/JOIN/GROUP BY). It reduces the data size to be loaded to CPU/RAM.<br />
<br />
Some HPC hardware supports PCIe switch between CPU and peripheral devices on PCIe bus. It also enables to bypass PCIe controller built-in CPU for SSD-to-GPU P2P DMA, if a pair of SSD and GPU are installed under the same PCIe-switch.<br />
From the standpoint of PostgreSQL, it makes sense that partition leaf accommodates bunch of SSDs for each PCIe-switch.<br />
Current version of partition-wise join allows JOIN between partitioned-tables with same distributed key, however, it makes sense to distribute and push down normal tables to individual partition-leafs, because we can run SQL on the peripheral device side.<br />
<br />
He suggested, 1) a lightweight hook on SetHintBit() to manage the heap blocks which are safe for P2P direct read, because GPU cannot access commit log, and zHeap may eventually remove the visibility-map infrastructure.<br />
2) an infrastructure to make a join path between normal-table and partitioned-table, without redundant inner read.<br />
<br />
From the audience, they comment on materialization is valuable for merge join also, not only hash-join. Even if inner table is distributed to 100 partition-leaf, shared materialized image prevents repeat of same scan. One other also comments the hashed/materialized table can be reduced if join-key contains partition-key and we can skip tuples obviously unmatch. One asked it is safe for non-equality join. KaiGai thinks it is equivalent to the current behavior for INNER JOIN, even if reorder GATHER and JOIN.<br />
One other question was about the hook on SetHintBit() - which information shall be delivered. KaiGai said that relation-id, block-number, iterm-pointer and informask, but it is detailed stuff.<br />
<br />
session slides are here:<br />
https://www.slideshare.net/kaigai/20181210-pgconfasia-unconference<br />
<br />
=== Built-In Job Scheduler ===<br />
<br />
Lack of scheduler in Postgres<br />
<br />
Extensions end up making their own scheduler, or require people to use cron<br />
<br />
Use cases<br />
<br />
* Partition rotation (pg_partman, Timescale)<br />
* Run-away queries and killing them<br />
* Time-based partial indexes (queries almost always look for the last one of something)<br />
* Manual vacuums<br />
<br />
In-core scheduler can also be useful for extensions to add recurring tasks, no need to make their own background worker<br />
<br />
Key features<br />
- RLS (Row Level Security) on user_name<br />
- Do NOT look like cron<br />
- Only execute stored procedures with a specific call signature<br />
<br />
We have stored procedures - no need for another program!<br />
<br />
<pre><br />
pg_job_schedule<br />
- job_name text UNIQUE<br />
- user_name text<br />
- procedure_name text/oid<br />
- initial_run tstz<br />
- run_frequency interval<br />
- last_run tstz<br />
- last_duration interval<br />
- last_successful_run tstz<br />
- last_successful_duration interval<br />
- max_run_duration interval<br />
- num_overlap int<br />
</pre><br />
<br />
Stored procedures are currently restricted in terms of languages, but from there you could call anything<br />
<br />
"How many backends do you use?"<br><br />
=> One scheduler job (background worker) that would then spawn separate workers for each invocation<br />
<br />
Dimitri: Should have a cap on the number of concurrent tasks that can be started<br />
<br />
Magnus: If you want long-running jobs, you might want to put priority on them<br />
<br />
Flag to run things on a primary/standby only (could also be solved with pg_is_in_recovery())<br />
<br />
What wouldn't work with this?<br><br />
=> VACUUM doesn't work, because it checks whether its a top-level statement<br><br />
=> CREATE INDEX CONCURRENTLY<br />
<br />
"Whats the value to only restricting it to execute stored procedures?"<br><br />
=> Simplicity was the main motivation<br><br />
=> One command text field is good too<br />
<br />
"Which databases is it going to run on?"<br><br />
=> Would need a way to specify the database<br />
<br />
"Are timezones a problem - how do you handle a day-light savings time change?"<br><br />
=> This should be taken care of by the schema as proposed<br />
<br />
"Isn't this similar to the ability to write a background worker without using C?"<br />
<br />
Magnus: "I want the ability to run a job now, without any other effects on the schedule"<br />
<br />
"Are last_duration / last_successful_run / etc necessary at all / should they be in a separate table?"<br><br />
=> Maybe should have "pg_stat_jobs_schedule" to store these and other things like rusage<br />
<br />
The core feature that could be using this is continuous materialized views<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2016_Developer_Unconference PgConf.ASIA 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2017_Developer_Unconference PgCon 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2017_Developer_Unconference PgConf.ASIA 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2018_Developer_Unconference PgCon 2018 Developer Unconference]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=IRC2RWNames&diff=29181IRC2RWNames2017-01-17T08:01:45Z<p>Rjuju: </p>
<hr />
<div>=== List of IRC nicks with their respective real world names ===<br />
<br />
You can find many PostgreSQL users and developers chatting in [irc://irc.freenode.net/postgresql #postgresql on freenode]. Here's more information about some of the regulars there. '''Note:''' people are on the list below only when they want to be. Do not (re-)add anyone without their express permission.<br />
<br />
{| border="1"<br />
|-<br />
!Nickname || Real Name<br />
|-<br />
|ads || Andreas Scherbaum<br />
|-<br />
|agliodbs, aglio2 (freenode), jberkus (oftc) || Josh Berkus<br />
|-<br />
|ahammond || Andrew Hammond<br />
|-<br />
|akretschmer || Andreas Kretschmer<br />
|-<br />
|alvherre || Alvaro Herrera<br />
|-<br />
|andres || Andres Freund<br />
|-<br />
|Assid || Satish Alwani<br />
|-<br />
|aurynn || Aurynn Shaw<br />
|-<br />
|beaud76 || Philippe Beaudoin<br />
|-<br />
|BlueAidan/BlueAidan_work || [[user:davidblewett | David Blewett]]<br />
|-<br />
|bmomjian || Bruce Momjian<br />
|-<br />
|cbbrowne || Christopher Browne<br />
|-<br />
|cce || Clark C. Evans<br />
|-<br />
|chicagoben || Benjamin Johnson<br />
|-<br />
|crab || Abhijit Menon-Sen<br />
|-<br />
|Crad || Gavin M. Roy<br />
|- <br />
|daamien || Damien Clochard<br />
|-<br />
|DarcyB || Darcy Buskermolen<br />
|-<br />
|darkixion || Thom Brown<br />
|-<br />
|davidfetter || David Fetter<br />
|-<br />
|dbb || Brian M Hamlin / darkblue_b<br />
|-<br />
|dcolish || [http://www.unencrypted.org Dan Colish]<br />
|-<br />
|dcramer || Dave Cramer<br />
|-<br />
|DeciBull, TheCougar || Jim C. Nasby<br />
|-<br />
|dennisb || Dennis Bj&ouml;rklund<br />
|-<br />
|depesz || Hubert Lubaczewski<br />
|-<br />
|devrimgunduz || Devrim G&uuml;nd&uuml;z<br />
|-<br />
|digicon || [http://digicondev.blogspot.com Zach Conrad]<br />
|-<br />
|digitalknight || Atri Sharma<br />
|-<br />
|dim || [http://tapoueh.org Dimitri Fontaine]<br />
|-<br />
|direvus || Brendan Jurd<br />
|-<br />
|drbair || Ryan Bair<br />
|-<br />
|DrLou || Lou Picciano<br />
|-<br />
|duck_tape || Adi Alurkar<br />
|-<br />
|dvl || [http://langille.org/ Dan Langille]<br />
|-<br />
|eggyknap || Joshua Tolley<br />
|-<br />
|endpoint_david || David Christensen<br />
|-<br />
|eulerto || Euler Taveira<br />
|-<br />
|f3ew/devdas || Devdas Vasu Bhagat<br />
|-<br />
|feivel || Michael Meskes<br />
|-<br />
|frost242 || Thomas Reiss<br />
|-<br />
|elein || Elein Mustain<br />
|-<br />
|Gibheer || Stefan Radomski<br />
|-<br />
|gleu || Guillaume Lelarge<br />
|-<br />
|gorthx || [[User:Gabrielle|Gabrielle Roth]]<br />
|-<br />
|grzm || Michael Glaesemann<br />
|-<br />
|gsmet || Guillaume Smet<br />
|-<br />
|gregs1104 || Greg Smith<br />
|-<br />
|gurjeet || [[User:singh.gurjeet|Gurjeet Singh]]<br />
|-<br />
|G_SabinoMullane || Greg Sabino Mullane<br />
|-<br />
|HarrisonF || Harrison Fisk<br />
|-<br />
|ioguix || Jehan-Guillaume de Rorthais<br />
|-<br />
|indigo || Phil Frost<br />
|-<br />
|intgr || Marti Raudsepp<br />
|-<br />
|JanniCash || Jan Wieck<br />
|-<br />
|jconway || Joe Conway<br />
|-<br />
|jdavis, jdavis_ || Jeff Davis<br />
|-<br />
|jkatz05 || Jonathan S. Katz<br />
|-<br />
|johto || Marko Tiikkaja<br />
|-<br />
|jurka || Kris Jurka<br />
|-<br />
|justatheory || David Wheeler<br />
|-<br />
|jpa || Jean-Paul Argudo<br />
|-<br />
|jwp || James Pye<br />
|-<br />
|j_williams || Josh Williams<br />
|-<br />
|keithf4 || [http://www.keithf4.com Keith Fiske]<br />
|-<br />
|kgrittn || Kevin Grittner<br />
|-<br />
|klando || [[User:c2main|Cédric Villemain]]<br />
|-<br />
|larryrtx || Larry Rosenman<br />
|-<br />
|linuxpoet, postgresman || Joshua D. Drake<br />
|-<br />
|lluad || Steve Atkins<br />
|-<br />
|lsmith || Lukas Smith<br />
|-<br />
|macdice || Thomas Munro<br />
|-<br />
|mage_ || Julien Cigar<br />
|-<br />
|magnush || Magnus Hagander<br />
|-<br />
|maiku41 || Mike Blackwell<br />
|-<br />
|marco44 || Marc Cousin<br />
|-<br />
|markwkm || Mark Wong<br />
|-<br />
|mastermind || [[user:mastermind | Stefan Kaltenbrunner]]<br />
|-<br />
|mbalmer || [[user:mbalmer | Marc Balmer]]<br />
|-<br />
|merlin83 || Chua Khee Chin<br />
|-<br />
|merlinm || Merlin Moncure<br />
|-<br />
|metatrontech || Chris Travers<br />
|-<br />
|miracee || Susanne Ebrecht<br />
|-<br />
|Moosbert || Peter Eisentraut<br />
|-<br />
|Myon || [[user:Myon | Christoph Berg]]<br />
|-<br />
|neilc || Neil Conway<br />
|-<br />
|oicu || Andrew Dunstan<br />
|-<br />
|okbobcz || Pavel Stehule<br />
|-<br />
|patryk || Patryk Kordylewski<br />
|-<br />
|pasha_golub || [http://pgolub.wordpress.com/ Pavel Golub]<br />
|-<br />
|pg_docbot || [[IRCBotSyntax]]<br />
|-<br />
|pgSnake || Dave Page<br />
|-<br />
|PJMODOS || Petr Jel&iacute;nek<br />
|-<br />
|Possible || Robert Ivens<br />
|-<br />
|postwait || Theo Schlossnagle<br />
|-<br />
|prothid || R Brenton Strickler<br />
|-<br />
|psoo || Bernd Helmle<br />
|-<br />
|PSUdaemon || Phil Sorber<br />
|-<br />
|pyarra || Philip Yarra<br />
|-<br />
|raptelan || [[user:Cshobe|Casey Allen Shobe]]<br />
|-<br />
|RhodiumToad (formerly AndrewSN) || Andrew Gierth<br />
|-<br />
|rjuju || Julien Rouhaud<br />
|-<br />
|Robe || [[user:Robe | Michael Renner]]<br />
|-<br />
|rotellaro || Federico Campoli<br />
|-<br />
|russ960 || [[user:russ960|Russ Johnson]]<br />
|-<br />
|rz || Kirill Simonov<br />
|-<br />
|saper || Marcin Cieślak<br />
|-<br />
|SAS || Stéphane Schildknecht<br />
|-<br />
|schmiddy || Josh Kupershmidt<br />
|-<br />
|scrappy || Marc G. Fournier<br />
|-<br />
|sehrope || Sehrope Sarkuni<br />
|-<br />
|selenamarie || Selena Deckelmann<br />
|-<br />
|SkippyDigits || Sherri Kalm<br />
|-<br />
|Snow-Man || Stephen Frost<br />
|-<br />
|Spritz || Matteo Beccati<br />
|-<br />
|sternocera || Peter Geoghegan<br />
|-<br />
|StuckMojo, MojoWork || Jon Erdman<br />
|-<br />
|swm || Gavin Sherry<br />
|-<br />
|vy || Volkan YAZICI<br />
|-<br />
|wulczer || Jan Urbański<br />
|-<br />
|xaprb || Baron Schwartz<br />
|-<br />
|xocolatl || Vik Fearing<br />
|-<br />
|xzilla, xzi11a || [[User:Xzilla|Robert Treat]]<br />
|}<br />
<br />
[[Category:Community]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Converting_from_other_Databases_to_PostgreSQL&diff=28775Converting from other Databases to PostgreSQL2016-12-10T18:20:41Z<p>Rjuju: </p>
<hr />
<div>== Non-specific ==<br />
<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines to PostgreSQL and back<br />
* [https://github.com/dimitri/pgloader pgloader] knows how to load data from MySQL, SQLite, MS SQL Server, dBase files, CSV files and fixed-width data files, and more. Released under The PostgreSQL Licence.<br />
* [https://dbconvert.com/postgresql/ DBConvert PostgreSQL database migration and sync software] Database conversion and synchronization between PostgreSQL/ Amazon RDS, MySQL, MS SQL Server, SQL Azure, Oracle, MS Access.<br />
* [http://www.easyfrom.net/ Converting data between PostgreSQL and others database formats] ESF Database Migration Toolkit enables you to transfer data across various databases, supporting PostgreSQL, MySQL, Oracle, SQL Server, IBM DB2, Informix, Microsoft Access, Microsoft Excel, dBase, Foxpro, Firbird, SQLite etc. - by Martin Williams<br />
* [http://troels.arvin.dk/db/rdbms/ Comparison of different SQL implementations] by Troels Arvin (covers PG 8.4 and MySQL 5.0)<br />
* [[Transactional DDL in PostgreSQL: A Competitive Analysis]] by Greg Smith<br />
* [[Migrating from one database to another with Pentaho ETL]] by Nicola Benaglia<br />
* [http://www.vive.net/products/datapro.htm dataPro] Conversion tool for PostgreSQL, SQLite, MySQL, Oracle, SQL Server and Microsoft Access. Transfer database objects between different databases and servers, convert tables schema and migrate data from one database type to another.<br />
* [http://www.datanamic.com/datadiff-crossdb/ DataDiff CrossDB] is a Windows GUI utility to compare and synchronize/transfer data from PostgreSQL to/from Oracle, MSSQL, MS Access or MySQL databases.<br />
* [http://www.sqlmaestro.com/products/postgresql/datawizard/ PostgreSQL Data Wizard] is a Windows GUI utility to transfer both schema and data from any ADO-compatible source (like MS Access, MySQL, SQL Server, Oracle, etc) to PostgreSQL.<br />
* [https://metacpan.org/module/SQL::Translator SQL::Translator] is a Perl module for translating table definitions between different software.<br />
* [[Foreign_data_wrappers|Foreign data wrappers]] may be useful for exporting data from other databases<br />
* [http://www.convert-in.com/pgskit.htm Postgres Migration Toolkit] Software pack to convert Oracle, MySQL, SQL Server and FoxPro to PostgreSQL, and vice versa.<br />
* [https://github.com/ei-grad/sqlacrossover sqlacrossover] SQLAlchemy-based cross-database migration tool<br />
* [https://skyvia.com Skyvia] Web service for cloud data integration for PostgreSQL with Salesforce, Dynamics CRM, SugarCRM, Zoho CRM, QuickBooks, FreshBooks, ExactTarget, MailChimp, Bigcommerce, MySQL, SQL Server, SQL Azure, Amazon RDS.<br />
* [https://github.com/OmniDB/OmniDB OmniDB] Open source web tool for database management and conversion. Currently supports PostgreSQL, Oracle, MariaDB, MySQL, Firebird, SQLite, MS Access, MS SQL Server, MS SQL Compact. OmniDB converts from any supported RDBMS to PostgreSQL and back.<br />
<br />
== Apache Derby ==<br />
<br />
* [https://github.com/billrobertson42/derby2pg derby2pg] Converter program that will convert the tables, data and indexes in a given schema to a script that can be used to populate a PostgreSQL database.<br />
<br />
== DBase II, III, IV+ / DBF Format ==<br />
<br />
* [http://www.convert-in.com/dbf2pgs.htm FoxPro-to-PostgreSQL] a program to migrate FoxPro databases to PostgreSQL server. The program does not use ODBC or any other middleware software. Command line support allows to script, automate and schedule the conversion process.<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from DBF (dBase, Clipper, Visual FoxPro and other DBF variants) to PostgreSQL and back<br />
* [https://github.com/dimitri/pgloader pgloader] by Dimitri Fontaine<br />
* [http://www.tv.com.pl/stepbystep/dbasepsql/ Convert .dbf files into PostgreSQL] by Tomasz Judycki<br />
* [[:Image:26.zip|A Visual Basic utility to convert Dbase III, IV, and V]] by Dennis Bazan<br />
* [[Porting data from dBASE IV to PostgreSQL]] by Vijay Deval (2002-09-07)<br />
* [https://github.com/kstrauser/pgdbf PgDBF : Simplified and optimized replacement for XBaseToPg] by Kirk Strauser<br />
<br />
== FileMaker Pro ==<br />
<br />
* [[Porting from FileMaker Pro to PostgreSQL]] by Michelle Murrain (24th August 2001)<br />
<br />
== IBM DB2 ==<br />
<br />
* [[:Image:DB2UDB-to-PG.pdf|Migrating from DB2 to PostgreSQL]]<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including IBM DB2, to PostgreSQL and back<br />
* [https://github.com/dalibo/db2topg/ db2topg] Migration tool to convert a DB2 UDB Database into a PostgreSQL database<br />
<br />
== Interbase ==<br />
<br />
* [https://github.com/kstrauser/dbreplicate DBReplicate - Simplify Interbase->PostgreSQL conversions] by Kirk Strauser<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including Interbase, to PostgreSQL and back<br />
<br />
== Microsoft Access ==<br />
<br />
* [http://www.postgresonline.com/journal/index.php?url=archives/346-Querying-MS-Access-and-other-ODBC-data-sources-with-OGR_FDW.html Linking to MS Access tables with ogr_fdw] If your PostgreSQL is on windows, you can use ogr_fdw foreign data wrapper, packaged with PostGIS 2.2+ bundle for windows via application stackbuilder. With PostgresQL 9.5, you can use IMPORT FOREIGN SCHEMA to link in all the MS access tables and then cherry pick how you want to restructure.<br />
* [http://docman.sourceforge.net/home_html/projects/sql/exportSQL3.txt exportSQL] - a Microsoft Access module which exports Access Database into MySQL, mSQL and PostgreSQL by Dobrica Pavlinusic. Based on the work of Pedro Freire<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including Access, to PostgreSQL and back<br />
* [http://mdbtools.sourceforge.net/ MDB Tools] by Brian Bruns<br />
** A quick way to dump all tables as tsv or csv files<br />
for TT in $(mdb-tables file.mdb); do<br />
mdb-export -Q -d '\t' -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.tsv"<br />
done<br />
<br />
for TT in $(mdb-tables file.mdb); do<br />
mdb-export -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.csv"<br />
done<br />
<br />
If the tablenames have embedded spaces...<br />
<br />
mdb-tables -1 file.mdb| while read TT<br />
do<br />
mdb-export -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.csv"<br />
done<br />
<br />
A shell script that may be useful for converting entire databases:<br />
<br />
#!/bin/sh -e<br />
<br />
mdbfn=$1<br />
schemafn=$2<br />
fkfn=$3<br />
datafn=$4<br />
schema=$5<br />
<br />
tf=$(tempfile)<br />
<br />
pre=""<br />
[ -n "${schema}" ] && pre="\"${schema}\"."<br />
<br />
mdb-schema "${mdbfn}" postgres > "${tf}"<br />
<br />
# Schema file<br />
echo "BEGIN;\n" > "${schemafn}"<br />
<br />
sp=""<br />
[ -n "${schema}" ] && echo "CREATE SCHEMA \"${schema}\";\n" >> "${schemafn}"<br />
[ -n "${schema}" ] && sp="SET search_path = \"${schema}\", pg_catalog;\n" <br />
<br />
echo ${sp} >> "${schemafn}"<br />
<br />
awk '($0 !~ /^ALTER TABLE.*FOREIGN KEY.*REFERENCES/) {print;}' "${tf}" >> "${schemafn}"<br />
<br />
echo "\nEND;" >> "${schemafn}"<br />
<br />
# Foreign keys file<br />
echo "BEGIN;\n" > "${fkfn}"<br />
echo ${sp} >> "${fkfn}"<br />
<br />
awk '($0 ~ /^ALTER TABLE.*FOREIGN KEY.*REFERENCES/) {print;}' "${tf}" >> "${fkfn}"<br />
<br />
echo "\nEND;" >> "${fkfn}"<br />
<br />
# Data file<br />
echo "BEGIN;\n" > "${datafn}"<br />
echo "SET CONSTRAINTS ALL DEFERRED;\n" >> "${datafn}"<br />
<br />
mdb-tables -1 "${mdbfn}" | while read TT<br />
do<br />
mdb-export -Q -d '\t' -D '%Y-%m-%d %H:%M:%S' "${mdbfn}" "$TT" > "${tf}"<br />
<br />
awk -v pre="${pre}" -v TT="${TT}" \<br />
'(NR==1) {gsub(/\t/,"\",\""); print "COPY " pre "\"" TT "\"(\"" $0 "\") FROM stdin;";}' "${tf}" >> "${datafn}"<br />
awk '(NR>1) {gsub(/\t\t/,"\t\\N\t"); gsub(/\t$/,"\t\\N"); gsub(/\t\t/,"\t\\N\t"); print;}' "${tf}" >> "${datafn}"<br />
<br />
echo "\\.\n" >> "${datafn}"<br />
done<br />
<br />
echo "END;" >> "${datafn}"<br />
<br />
rm -f "${tf}"<br />
<br />
If this script is saved to the file access2psql.sh and made executable, then it would be used as follows:<br />
<br />
access2psql.sh file.mdb schema.sql foreignkeys.sql data.sql pg_schema_name<br />
psql -f schema.sql pg_db_name<br />
psql -f data.sql pg_db_name<br />
psql -f foreignkeys.sql pg_db_name<br />
<br />
This script won't work properly if there are tab characters in text columns, though the call to mdb-export could be modified to export INSERT statements to fix this. Also, mdb-schema has trouble representing multi-column foreign keys, so foreignkeys.sql may need some manual editing.<br />
<br />
* [http://convertdb.com/access/postgresql/ Export Microsoft Access to PostgreSQL. Cross database synchronization]<br />
* [http://www.olschimke.eu/2012/08/07/importing-microsoft-access-mdb-into-postgresql-on-linux-postgres/ Importing Microsoft Access MDB into PostgreSQL on Linux] by Michael Olschimke<br />
* [[Microsoft Access to PostgreSQL Conversion]] by Jon Hutchings (2001-07-20)<br />
* [http://pgfoundry.org/projects/access2pgsql/ access2pgsql tool] by Mariano Reingart<br />
* [http://www.postgresonline.com/journal/archives/24-Using-MS-Access-with-PostgreSQL.html Using MS Access as front end and PostgreSQL as back-end database] by PostgresOnLine Journal - 2008-01-31<br />
<br />
== Microsoft SQL Server ==<br />
<br />
* [http://www.convert-in.com/mss2pgs.htm MSSQL-to-PostgreSQL] is a migration utility to convert SQL Server or SQL Azure databases to PostgreSQL. Option to filter data using SELECT-queries, synchronization mode, command line support.<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including SQL Server, to PostgreSQL and back<br />
* [http://dalibo.github.io/sqlserver2pgsql/ sqlserver2pgsql] Migration tool to convert a Microsoft SQL Server Database into a PostgreSQL database<br />
* [https://github.com/dimitri/pgloader pgloader] by Dimitri Fontaine<br />
* [https://dbconvert.com/mssql/postgresql/ DBConvert for PostgreSQL and MS SQL Server] Conversion and Sync software solutions for cross database migration between PostgreSQL/ Amazon RDS and MS SQL Server/ SQL Azure<br />
* [http://sqlblog.com/blogs/alexander_kuznetsov/archive/tags/PostgreSql/default.aspx Learning PostgreSQL (Alexander Kuznetsov)] - series of blog articles for SQL Server users (2013-10 ~ 2013-11)<br />
* [http://www.postgresonline.com/journal/archives/219-SQL-Server-to-PostgreSQL-Converting-table-structure.html Converting SQL Server Table Structure to PostgreSQL] Leo Hsu and Regina Obe (2011-09-03)<br />
* [[:Image:5.pdf|Conversion of Microsoft SQL/ASP applications to PostgreSQL]] by Ethan Townsend (2005-06-23)<br />
* [[Converting your data from MS SQL Server 7 to PostgreSQL 7.1.x]] by Ryan C. Bonham (2002-01-05)<br />
* [[Microsoft SQL Server to PostgreSQL Migration by Ian Harding]] (2001-09-17)<br />
* [http://www.bostongis.com/PrinterFriendly.aspx?content_name=sqlserver2008r2_oracle11gr2_postgis15_compare Compare SQL Server 2008 R2, Oracle 11G R2, PostgreSQL/PostGIS 1.5 Spatial Features] BostonGIS (2010-06-01) <br />
* [http://www.postgresonline.com/journal/index.php?/archives/51-Cross-Compare-of-SQL-Server,-MySQL,-and-PostgreSQL.html Cross Compare of SQL Server 2005, MySQL 5, and PostgreSQL 8.3] by Leo Hsu and Regina Obe (2008-05-13)<br />
* [http://www.postgresonline.com/journal/index.php?/archives/130-Cross-Compare-of-PostgreSQL-8.4,-SQL-Server-2008,-MySQL-5.1.html Cross Compare of SQL Server 2008, MySQL 5.1, and PostgreSQL 8.4] by Leo Hsu and Regina Obe (2009-08-15)<br />
* [http://www.postgresonline.com/journal/archives/179-Universal-Unique-Identifiers-PostgreSQL-SQL-Server-Compare.html Universal Unique Identifiers PostgreSQL SQL Server Compare] Demonstrates how to implement SQL Server NEWID() GUID uniqueidentifier type in PostgreSQL by Leo Hsu and Regina Obe (2010-10-07)<br />
<br />
== MySQL ==<br />
<br />
<br />
=== Scripts, programs ===<br />
<br />
==== 2015 ====<br />
* [http://www.convert-in.com/sql2pgs.htm MySQL-to-PostgreSQL] is a program to migrate MySQL databases to PostgreSQL server. Option to filter data using SELECT-queries, synchronization mode, command line support.<br />
* [https://github.com/AnatolyUss/FromMySqlToPostgreSql FromMySqlToPostgreSql migration tool by Anatoly Khaytovich], provides an accurate migration of table data, indices, PKs, FKs... Makes an extensive use of PostgreSQL COPY protocol.<br />
<br />
==== 2014 ====<br />
<br />
* [https://github.com/mihailShumilov/mysql2postgresql MySQL/PostgreSQL Converter from Mihail Shumilov], written on PHP. Supported big DB<br />
<br />
==== 2013 ====<br />
* [https://github.com/lanyrd/mysql-postgresql-converter MySQL/PostgreSQL Converter from Lanyrd], based on [http://lanyrd.com/blog/2012/lanyrds-big-move/ work detailed here]<br />
* [https://github.com/dimitri/pgloader pgloader], from Dimitri Fontaine, based on [http://tapoueh.org/blog/2013/01/28-pgloader-future work detailed here] and [http://tapoueh.org/blog/2013/11/12-migrating-sakila Migrating Sakila from MySQL to PostgreSQL]<br />
* [https://github.com/philipsoutham/py-mysql2pgsql py-mysql2pgsql], [https://github.com/mozilla/airmozilla/pull/61/files#diff-30 example use case here]<br />
* [https://github.com/bdigital/my_etl MyETL], Tool that works with the [https://github.com/bdigital/mysql_fdw MySQL FDW]<br />
<br />
==== Previously ====<br />
* [http://www.enterprisedb.com/products/download.do Free MySQL to Postgres Migration Wizard by EnterpriseDB v1.1] by EnterpriseDB Corporation. Available without registration via PostgreSQL installer wizard.<br />
* [http://www.pgsql.com/download/ Conversion tool for migrating from MySQL to PostgreSQL] by PostgreSQL Inc.<br />
* [http://en.wikibooks.org/wiki/Converting_MySQL_to_PostgreSQL Converting MySQL to PostgreSQL] - from en.wikibooks.org<br />
* [http://convertdb.com/mysql/postgresql/ Migrate MySQL to PostgreSQL.] - Cross database bidirectional synchronization<br />
* [http://www.sourcefiles.org/Databases/Utilities/Conversion/my2pg.pl my2pg.pl] - A Perl script used to convert a MySQL database dump to PostgreSQL-compatible format, by Maxim Rudensky and Valentine Danilchuk<br />
* [http://pgfoundry.org/projects/mysql2pgsql/ mysql2pgsql] - A Perl script used to convert MySQL databases dump to a PostgreSQL-compatible format<br />
* [http://mp2p.mikekohn.net/ MySQL PHP to PostgreSQL] by Michael Kohn<br />
* [http://www.gab.lc/script_php_my2pg.php PHP_my2pg] PHP script by Gabriel Bordeaux<br />
* [http://tryolabs.com/Blog/2012/02/10/django-migrating-mysql-postgresql/ Migrating from MySQL to Postgresql with Django] blogpost by alejandro<br />
* [http://github.com/maxlapshin/mysql2postgres Migrates MySQL to PostgreSQL] Ruby script by Max Lapshin<br />
# To install mysql2psql (under ubuntu 11.10): No need to get from github, just:<br />
sudo apt-get install ruby gems libmysqlclient-dev libpq-dev<br />
gem install mysql pg mysql2psql<br />
# To get info about the mysql socket:<br />
netstat -l | grep mysql<br />
mysql2psql # creates a .yml templae<br />
vi mysql2psql.yml # edit the template<br />
mysql2psql # connects to mysql database and write into postgres database<br />
<br />
* [http://pypi.python.org/pypi/py-mysql2pgsql py-mysql2pgsql] Python script to convert MySQL database to PostgreSQL compatible file or export directly to a PostgreSQL server<br />
* [https://bitbucket.org/iceone/mysql2pgsql/ my2pg.py] MySQL to PostgreSQL database conversion (not the same as above)<br />
* [http://phpseclib.sourceforge.net/sql/ SQL Data Definition Language (DDL) Conversion to MySQL, PostgreSQL and MS-SQL] by phpBB Group<br />
<br />
=== Documentation ===<br />
<br />
* [http://www.sitepoint.com/article/site-mysql-postgresql-1 Backend Database Switcheroo Howto Part I] by Nathan Matias<br />
* [http://www.sitepoint.com/article/site-mysql-postgresql-2 Backend Database Switcheroo Howto Part II] by Nathan Matias<br />
* [http://www.xach.com/aolserver/mysql-to-postgresql.html How-To: Migrating from MySQL to PostgreSQL] by Zach Beane<br />
* [[Things to find out about when moving from MySQL to PostgreSQL]] by Joel Burton (8th April 2001)<br />
* [[Why PostgreSQL Instead of MySQL: Comparing Reliability and Speed in 2007]] by Greg Smith<br />
* [http://www.postgresonline.com/journal/index.php?/archives/51-Cross-Compare-of-SQL-Server,-MySQL,-and-PostgreSQL.html Cross Compare of SQL Server, MySQL, and PostgreSQL] by Leo Hsu and Regina Obe (2008-05-13)<br />
* [[How to make a proper migration from MySQL to PostgreSQL]]<br />
* [http://andreas.scherbaum.la/writings/migrate_web_platforms.pdf Migrating a community platform from Mysql to PostgreSQL] by Andreas Scherbaum (Prato, 2007)<br />
* [http://okbob.blogspot.com/2009/08/mysql-functions-for-postgresql.html MySQL functions for PostgreSQL] by Pavel Stěhule<br />
<br />
== Oracle ==<br />
<br />
=== Utilities, tools, scripts etc. ===<br />
<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines to PostgreSQL and back<br />
* [https://dbconvert.com/oracle/postgresql/ Database migration for Oracle and PostgreSQL] Software for cross - database conversion and sync between Oracle and PostgreSQL.<br />
* [http://ora2pg.darold.net/ Ora2Pg - Oracle to PostgreSQL database schema converter] by Gilles Darold<br />
* [http://oracle-fdw.projects.pgfoundry.org/ PostgreSQL Foreign Data Wrapper for Oracle (oracle_fdw)] - an [[Fdw|FDW]] providing support to access Oracle databases from within PostgreSQL<br />
* [http://orafce.projects.pgfoundry.org/ Orafce] - implements common Oracle functions in PostgreSQL for compatibility<br />
* [http://www.convert-in.com/ora2pgs.htm Oracle-to-PostgreSQL] - a program to migrate Oracle databases to PostgreSQL server. The program has high performance due to direct connection to data source and destination databases (it does not use ODBC or any other middleware software). Command line support allows to script, automate and schedule the conversion process.<br />
<br />
=== Documentation, articles, presentations etc. ===<br />
<br />
==== PostgreSQL documentation ====<br />
* [http://www.postgresql.org/docs/current/interactive/plpgsql-porting.html Porting from Oracle PL/SQL]<br />
<br />
==== Articles and presentations (in reverse chronological order) ====<br />
<br />
* [http://tapoueh.org/confs/2013/10/29-pgconfeu-2013 From MySQL to PostgreSQL with pgloader] (English) talk at PostgreSQL Conference Europe, Dimitri Fontaine, 2013-10-29<br />
* [http://wiki.cenatic.es/wikiesp/index.php/Procedimiento_de_An%C3%A1lisis_y_Planificaci%C3%B3n_de_Migraci%C3%B3n_de_Oracle_10g_R2_a_PostgreSQL_9.1.3 Analysis and Planning Process for Migration from Oracle 10g R2 to PostgreSQL 9.1.3] (Spanish) by the Spanish National Competence Center for the Application of Opensource Technologies [http://www.cenatic.es (CENATIC)] (2012-06)<br />
* [http://www.bostongis.com/PrinterFriendly.aspx?content_name=sqlserver2008r2_oracle11gr2_postgis15_compare Compare SQL Server 2008 R2, Oracle 11G R2, PostgreSQL/PostGIS 1.5 Spatial Features] BostonGIS (2010-06-01)<br />
* [http://wiki.kandalaya.org/cgi-bin/twiki/view/Main/MigratingOracleToPostgreSQL Migrating Oracle Databases to PostgreSQL using Xen on Debian GNU/Linux] by Raj Mathur (Kandalaya) (2008-12-26)<br />
* [http://www.pgcon.org/2008/schedule/track/Tutorial/62.en.html Porting Oracle Applications to PostgreSQL] by Peter Eisentraut (PGCon 2008)<br />
* [[PostgreSQL for Oracle DBAs]] by Richard Stephan (2007-06-01)<br />
* [[:Image:Pg 8.1 J2EE v1.0.pdf|Porting JDBC applications from Oracle to PostgreSQL]] by Chris Drawater (2006-03-24)<br />
** [http://www.holindis.co.uk/ Related documents for J2EE, Tomcat, and Oracle migrations] by Chris Drawater (2007-01-15)<br />
* [[Interview with Mark Stosberg (From Oracle/tcl to PostgreSQL/Perl)]] by Robert Treat (2004-10-24)<br />
* [[Oracle to Postgres Conversion]] by James Shannon, Ben Adida, and Don Baccus<br />
<br />
== Progress RDBMS ==<br />
<br />
* [http://knol.google.com/k/tomasz-judycki/convert-progress-rdbms-to-postgresql/2rifd79ifooa9/4# Convert Progress RDBMS to PostgreSQL] by Tomasz Judycki<br />
<br />
== Converting PostgreSQL Databases to other Databases ==<br />
<br />
* [http://docman.sourceforge.net/home_html/projects/sql/pgsql2interbase PostgreSQL to InterBase dump file converter] by Dobrica Pavlinusic<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from PostgreSQL to more than 30 database engines<br />
* [https://github.com/OmniDB/OmniDB OmniDB] Open source web tool for database management and conversion. Currently supports PostgreSQL, Oracle, MariaDB, MySQL, Firebird, SQLite, MS Access, MS SQL Server, MS SQL Compact. OmniDB converts from any supported RDBMS to PostgreSQL and back.<br />
<br />
[[Category:Install]]<br />
[[Category:General articles and guides]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Converting_from_other_Databases_to_PostgreSQL&diff=28774Converting from other Databases to PostgreSQL2016-12-10T18:18:34Z<p>Rjuju: add derby2pg</p>
<hr />
<div>== Non-specific ==<br />
<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines to PostgreSQL and back<br />
* [https://github.com/dimitri/pgloader pgloader] knows how to load data from MySQL, SQLite, MS SQL Server, dBase files, CSV files and fixed-width data files, and more. Released under The PostgreSQL Licence.<br />
* [https://dbconvert.com/postgresql/ DBConvert PostgreSQL database migration and sync software] Database conversion and synchronization between PostgreSQL/ Amazon RDS, MySQL, MS SQL Server, SQL Azure, Oracle, MS Access.<br />
* [http://www.easyfrom.net/ Converting data between PostgreSQL and others database formats] ESF Database Migration Toolkit enables you to transfer data across various databases, supporting PostgreSQL, MySQL, Oracle, SQL Server, IBM DB2, Informix, Microsoft Access, Microsoft Excel, dBase, Foxpro, Firbird, SQLite etc. - by Martin Williams<br />
* [http://troels.arvin.dk/db/rdbms/ Comparison of different SQL implementations] by Troels Arvin (covers PG 8.4 and MySQL 5.0)<br />
* [[Transactional DDL in PostgreSQL: A Competitive Analysis]] by Greg Smith<br />
* [[Migrating from one database to another with Pentaho ETL]] by Nicola Benaglia<br />
* [http://www.vive.net/products/datapro.htm dataPro] Conversion tool for PostgreSQL, SQLite, MySQL, Oracle, SQL Server and Microsoft Access. Transfer database objects between different databases and servers, convert tables schema and migrate data from one database type to another.<br />
* [http://www.datanamic.com/datadiff-crossdb/ DataDiff CrossDB] is a Windows GUI utility to compare and synchronize/transfer data from PostgreSQL to/from Oracle, MSSQL, MS Access or MySQL databases.<br />
* [http://www.sqlmaestro.com/products/postgresql/datawizard/ PostgreSQL Data Wizard] is a Windows GUI utility to transfer both schema and data from any ADO-compatible source (like MS Access, MySQL, SQL Server, Oracle, etc) to PostgreSQL.<br />
* [https://metacpan.org/module/SQL::Translator SQL::Translator] is a Perl module for translating table definitions between different software.<br />
* [[Foreign_data_wrappers|Foreign data wrappers]] may be useful for exporting data from other databases<br />
* [http://www.convert-in.com/pgskit.htm Postgres Migration Toolkit] Software pack to convert Oracle, MySQL, SQL Server and FoxPro to PostgreSQL, and vice versa.<br />
* [https://github.com/ei-grad/sqlacrossover sqlacrossover] SQLAlchemy-based cross-database migration tool<br />
* [https://skyvia.com Skyvia] Web service for cloud data integration for PostgreSQL with Salesforce, Dynamics CRM, SugarCRM, Zoho CRM, QuickBooks, FreshBooks, ExactTarget, MailChimp, Bigcommerce, MySQL, SQL Server, SQL Azure, Amazon RDS.<br />
* [https://github.com/OmniDB/OmniDB OmniDB] Open source web tool for database management and conversion. Currently supports PostgreSQL, Oracle, MariaDB, MySQL, Firebird, SQLite, MS Access, MS SQL Server, MS SQL Compact. OmniDB converts from any supported RDBMS to PostgreSQL and back.<br />
<br />
== Apache Derby ==<br />
<br />
* [[https://github.com/billrobertson42/derby2pg derby2pg]] Converter program that will convert the tables, data and indexes in a given schema to a script that can be used to populate a PostgreSQL database.<br />
<br />
== DBase II, III, IV+ / DBF Format ==<br />
<br />
* [http://www.convert-in.com/dbf2pgs.htm FoxPro-to-PostgreSQL] a program to migrate FoxPro databases to PostgreSQL server. The program does not use ODBC or any other middleware software. Command line support allows to script, automate and schedule the conversion process.<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from DBF (dBase, Clipper, Visual FoxPro and other DBF variants) to PostgreSQL and back<br />
* [https://github.com/dimitri/pgloader pgloader] by Dimitri Fontaine<br />
* [http://www.tv.com.pl/stepbystep/dbasepsql/ Convert .dbf files into PostgreSQL] by Tomasz Judycki<br />
* [[:Image:26.zip|A Visual Basic utility to convert Dbase III, IV, and V]] by Dennis Bazan<br />
* [[Porting data from dBASE IV to PostgreSQL]] by Vijay Deval (2002-09-07)<br />
* [https://github.com/kstrauser/pgdbf PgDBF : Simplified and optimized replacement for XBaseToPg] by Kirk Strauser<br />
<br />
== FileMaker Pro ==<br />
<br />
* [[Porting from FileMaker Pro to PostgreSQL]] by Michelle Murrain (24th August 2001)<br />
<br />
== IBM DB2 ==<br />
<br />
* [[:Image:DB2UDB-to-PG.pdf|Migrating from DB2 to PostgreSQL]]<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including IBM DB2, to PostgreSQL and back<br />
* [[https://github.com/dalibo/db2topg/ db2topg]] Migration tool to convert a DB2 UDB Database into a PostgreSQL database<br />
<br />
== Interbase ==<br />
<br />
* [https://github.com/kstrauser/dbreplicate DBReplicate - Simplify Interbase->PostgreSQL conversions] by Kirk Strauser<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including Interbase, to PostgreSQL and back<br />
<br />
== Microsoft Access ==<br />
<br />
* [http://www.postgresonline.com/journal/index.php?url=archives/346-Querying-MS-Access-and-other-ODBC-data-sources-with-OGR_FDW.html Linking to MS Access tables with ogr_fdw] If your PostgreSQL is on windows, you can use ogr_fdw foreign data wrapper, packaged with PostGIS 2.2+ bundle for windows via application stackbuilder. With PostgresQL 9.5, you can use IMPORT FOREIGN SCHEMA to link in all the MS access tables and then cherry pick how you want to restructure.<br />
* [http://docman.sourceforge.net/home_html/projects/sql/exportSQL3.txt exportSQL] - a Microsoft Access module which exports Access Database into MySQL, mSQL and PostgreSQL by Dobrica Pavlinusic. Based on the work of Pedro Freire<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including Access, to PostgreSQL and back<br />
* [http://mdbtools.sourceforge.net/ MDB Tools] by Brian Bruns<br />
** A quick way to dump all tables as tsv or csv files<br />
for TT in $(mdb-tables file.mdb); do<br />
mdb-export -Q -d '\t' -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.tsv"<br />
done<br />
<br />
for TT in $(mdb-tables file.mdb); do<br />
mdb-export -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.csv"<br />
done<br />
<br />
If the tablenames have embedded spaces...<br />
<br />
mdb-tables -1 file.mdb| while read TT<br />
do<br />
mdb-export -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.csv"<br />
done<br />
<br />
A shell script that may be useful for converting entire databases:<br />
<br />
#!/bin/sh -e<br />
<br />
mdbfn=$1<br />
schemafn=$2<br />
fkfn=$3<br />
datafn=$4<br />
schema=$5<br />
<br />
tf=$(tempfile)<br />
<br />
pre=""<br />
[ -n "${schema}" ] && pre="\"${schema}\"."<br />
<br />
mdb-schema "${mdbfn}" postgres > "${tf}"<br />
<br />
# Schema file<br />
echo "BEGIN;\n" > "${schemafn}"<br />
<br />
sp=""<br />
[ -n "${schema}" ] && echo "CREATE SCHEMA \"${schema}\";\n" >> "${schemafn}"<br />
[ -n "${schema}" ] && sp="SET search_path = \"${schema}\", pg_catalog;\n" <br />
<br />
echo ${sp} >> "${schemafn}"<br />
<br />
awk '($0 !~ /^ALTER TABLE.*FOREIGN KEY.*REFERENCES/) {print;}' "${tf}" >> "${schemafn}"<br />
<br />
echo "\nEND;" >> "${schemafn}"<br />
<br />
# Foreign keys file<br />
echo "BEGIN;\n" > "${fkfn}"<br />
echo ${sp} >> "${fkfn}"<br />
<br />
awk '($0 ~ /^ALTER TABLE.*FOREIGN KEY.*REFERENCES/) {print;}' "${tf}" >> "${fkfn}"<br />
<br />
echo "\nEND;" >> "${fkfn}"<br />
<br />
# Data file<br />
echo "BEGIN;\n" > "${datafn}"<br />
echo "SET CONSTRAINTS ALL DEFERRED;\n" >> "${datafn}"<br />
<br />
mdb-tables -1 "${mdbfn}" | while read TT<br />
do<br />
mdb-export -Q -d '\t' -D '%Y-%m-%d %H:%M:%S' "${mdbfn}" "$TT" > "${tf}"<br />
<br />
awk -v pre="${pre}" -v TT="${TT}" \<br />
'(NR==1) {gsub(/\t/,"\",\""); print "COPY " pre "\"" TT "\"(\"" $0 "\") FROM stdin;";}' "${tf}" >> "${datafn}"<br />
awk '(NR>1) {gsub(/\t\t/,"\t\\N\t"); gsub(/\t$/,"\t\\N"); gsub(/\t\t/,"\t\\N\t"); print;}' "${tf}" >> "${datafn}"<br />
<br />
echo "\\.\n" >> "${datafn}"<br />
done<br />
<br />
echo "END;" >> "${datafn}"<br />
<br />
rm -f "${tf}"<br />
<br />
If this script is saved to the file access2psql.sh and made executable, then it would be used as follows:<br />
<br />
access2psql.sh file.mdb schema.sql foreignkeys.sql data.sql pg_schema_name<br />
psql -f schema.sql pg_db_name<br />
psql -f data.sql pg_db_name<br />
psql -f foreignkeys.sql pg_db_name<br />
<br />
This script won't work properly if there are tab characters in text columns, though the call to mdb-export could be modified to export INSERT statements to fix this. Also, mdb-schema has trouble representing multi-column foreign keys, so foreignkeys.sql may need some manual editing.<br />
<br />
* [http://convertdb.com/access/postgresql/ Export Microsoft Access to PostgreSQL. Cross database synchronization]<br />
* [http://www.olschimke.eu/2012/08/07/importing-microsoft-access-mdb-into-postgresql-on-linux-postgres/ Importing Microsoft Access MDB into PostgreSQL on Linux] by Michael Olschimke<br />
* [[Microsoft Access to PostgreSQL Conversion]] by Jon Hutchings (2001-07-20)<br />
* [http://pgfoundry.org/projects/access2pgsql/ access2pgsql tool] by Mariano Reingart<br />
* [http://www.postgresonline.com/journal/archives/24-Using-MS-Access-with-PostgreSQL.html Using MS Access as front end and PostgreSQL as back-end database] by PostgresOnLine Journal - 2008-01-31<br />
<br />
== Microsoft SQL Server ==<br />
<br />
* [http://www.convert-in.com/mss2pgs.htm MSSQL-to-PostgreSQL] is a migration utility to convert SQL Server or SQL Azure databases to PostgreSQL. Option to filter data using SELECT-queries, synchronization mode, command line support.<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including SQL Server, to PostgreSQL and back<br />
* [[http://dalibo.github.io/sqlserver2pgsql/ sqlserver2pgsql]] Migration tool to convert a Microsoft SQL Server Database into a PostgreSQL database<br />
* [https://github.com/dimitri/pgloader pgloader] by Dimitri Fontaine<br />
* [https://dbconvert.com/mssql/postgresql/ DBConvert for PostgreSQL and MS SQL Server] Conversion and Sync software solutions for cross database migration between PostgreSQL/ Amazon RDS and MS SQL Server/ SQL Azure<br />
* [http://sqlblog.com/blogs/alexander_kuznetsov/archive/tags/PostgreSql/default.aspx Learning PostgreSQL (Alexander Kuznetsov)] - series of blog articles for SQL Server users (2013-10 ~ 2013-11)<br />
* [http://www.postgresonline.com/journal/archives/219-SQL-Server-to-PostgreSQL-Converting-table-structure.html Converting SQL Server Table Structure to PostgreSQL] Leo Hsu and Regina Obe (2011-09-03)<br />
* [[:Image:5.pdf|Conversion of Microsoft SQL/ASP applications to PostgreSQL]] by Ethan Townsend (2005-06-23)<br />
* [[Converting your data from MS SQL Server 7 to PostgreSQL 7.1.x]] by Ryan C. Bonham (2002-01-05)<br />
* [[Microsoft SQL Server to PostgreSQL Migration by Ian Harding]] (2001-09-17)<br />
* [http://www.bostongis.com/PrinterFriendly.aspx?content_name=sqlserver2008r2_oracle11gr2_postgis15_compare Compare SQL Server 2008 R2, Oracle 11G R2, PostgreSQL/PostGIS 1.5 Spatial Features] BostonGIS (2010-06-01) <br />
* [http://www.postgresonline.com/journal/index.php?/archives/51-Cross-Compare-of-SQL-Server,-MySQL,-and-PostgreSQL.html Cross Compare of SQL Server 2005, MySQL 5, and PostgreSQL 8.3] by Leo Hsu and Regina Obe (2008-05-13)<br />
* [http://www.postgresonline.com/journal/index.php?/archives/130-Cross-Compare-of-PostgreSQL-8.4,-SQL-Server-2008,-MySQL-5.1.html Cross Compare of SQL Server 2008, MySQL 5.1, and PostgreSQL 8.4] by Leo Hsu and Regina Obe (2009-08-15)<br />
* [http://www.postgresonline.com/journal/archives/179-Universal-Unique-Identifiers-PostgreSQL-SQL-Server-Compare.html Universal Unique Identifiers PostgreSQL SQL Server Compare] Demonstrates how to implement SQL Server NEWID() GUID uniqueidentifier type in PostgreSQL by Leo Hsu and Regina Obe (2010-10-07)<br />
<br />
== MySQL ==<br />
<br />
<br />
=== Scripts, programs ===<br />
<br />
==== 2015 ====<br />
* [http://www.convert-in.com/sql2pgs.htm MySQL-to-PostgreSQL] is a program to migrate MySQL databases to PostgreSQL server. Option to filter data using SELECT-queries, synchronization mode, command line support.<br />
* [https://github.com/AnatolyUss/FromMySqlToPostgreSql FromMySqlToPostgreSql migration tool by Anatoly Khaytovich], provides an accurate migration of table data, indices, PKs, FKs... Makes an extensive use of PostgreSQL COPY protocol.<br />
<br />
==== 2014 ====<br />
<br />
* [https://github.com/mihailShumilov/mysql2postgresql MySQL/PostgreSQL Converter from Mihail Shumilov], written on PHP. Supported big DB<br />
<br />
==== 2013 ====<br />
* [https://github.com/lanyrd/mysql-postgresql-converter MySQL/PostgreSQL Converter from Lanyrd], based on [http://lanyrd.com/blog/2012/lanyrds-big-move/ work detailed here]<br />
* [https://github.com/dimitri/pgloader pgloader], from Dimitri Fontaine, based on [http://tapoueh.org/blog/2013/01/28-pgloader-future work detailed here] and [http://tapoueh.org/blog/2013/11/12-migrating-sakila Migrating Sakila from MySQL to PostgreSQL]<br />
* [https://github.com/philipsoutham/py-mysql2pgsql py-mysql2pgsql], [https://github.com/mozilla/airmozilla/pull/61/files#diff-30 example use case here]<br />
* [https://github.com/bdigital/my_etl MyETL], Tool that works with the [https://github.com/bdigital/mysql_fdw MySQL FDW]<br />
<br />
==== Previously ====<br />
* [http://www.enterprisedb.com/products/download.do Free MySQL to Postgres Migration Wizard by EnterpriseDB v1.1] by EnterpriseDB Corporation. Available without registration via PostgreSQL installer wizard.<br />
* [http://www.pgsql.com/download/ Conversion tool for migrating from MySQL to PostgreSQL] by PostgreSQL Inc.<br />
* [http://en.wikibooks.org/wiki/Converting_MySQL_to_PostgreSQL Converting MySQL to PostgreSQL] - from en.wikibooks.org<br />
* [http://convertdb.com/mysql/postgresql/ Migrate MySQL to PostgreSQL.] - Cross database bidirectional synchronization<br />
* [http://www.sourcefiles.org/Databases/Utilities/Conversion/my2pg.pl my2pg.pl] - A Perl script used to convert a MySQL database dump to PostgreSQL-compatible format, by Maxim Rudensky and Valentine Danilchuk<br />
* [http://pgfoundry.org/projects/mysql2pgsql/ mysql2pgsql] - A Perl script used to convert MySQL databases dump to a PostgreSQL-compatible format<br />
* [http://mp2p.mikekohn.net/ MySQL PHP to PostgreSQL] by Michael Kohn<br />
* [http://www.gab.lc/script_php_my2pg.php PHP_my2pg] PHP script by Gabriel Bordeaux<br />
* [http://tryolabs.com/Blog/2012/02/10/django-migrating-mysql-postgresql/ Migrating from MySQL to Postgresql with Django] blogpost by alejandro<br />
* [http://github.com/maxlapshin/mysql2postgres Migrates MySQL to PostgreSQL] Ruby script by Max Lapshin<br />
# To install mysql2psql (under ubuntu 11.10): No need to get from github, just:<br />
sudo apt-get install ruby gems libmysqlclient-dev libpq-dev<br />
gem install mysql pg mysql2psql<br />
# To get info about the mysql socket:<br />
netstat -l | grep mysql<br />
mysql2psql # creates a .yml templae<br />
vi mysql2psql.yml # edit the template<br />
mysql2psql # connects to mysql database and write into postgres database<br />
<br />
* [http://pypi.python.org/pypi/py-mysql2pgsql py-mysql2pgsql] Python script to convert MySQL database to PostgreSQL compatible file or export directly to a PostgreSQL server<br />
* [https://bitbucket.org/iceone/mysql2pgsql/ my2pg.py] MySQL to PostgreSQL database conversion (not the same as above)<br />
* [http://phpseclib.sourceforge.net/sql/ SQL Data Definition Language (DDL) Conversion to MySQL, PostgreSQL and MS-SQL] by phpBB Group<br />
<br />
=== Documentation ===<br />
<br />
* [http://www.sitepoint.com/article/site-mysql-postgresql-1 Backend Database Switcheroo Howto Part I] by Nathan Matias<br />
* [http://www.sitepoint.com/article/site-mysql-postgresql-2 Backend Database Switcheroo Howto Part II] by Nathan Matias<br />
* [http://www.xach.com/aolserver/mysql-to-postgresql.html How-To: Migrating from MySQL to PostgreSQL] by Zach Beane<br />
* [[Things to find out about when moving from MySQL to PostgreSQL]] by Joel Burton (8th April 2001)<br />
* [[Why PostgreSQL Instead of MySQL: Comparing Reliability and Speed in 2007]] by Greg Smith<br />
* [http://www.postgresonline.com/journal/index.php?/archives/51-Cross-Compare-of-SQL-Server,-MySQL,-and-PostgreSQL.html Cross Compare of SQL Server, MySQL, and PostgreSQL] by Leo Hsu and Regina Obe (2008-05-13)<br />
* [[How to make a proper migration from MySQL to PostgreSQL]]<br />
* [http://andreas.scherbaum.la/writings/migrate_web_platforms.pdf Migrating a community platform from Mysql to PostgreSQL] by Andreas Scherbaum (Prato, 2007)<br />
* [http://okbob.blogspot.com/2009/08/mysql-functions-for-postgresql.html MySQL functions for PostgreSQL] by Pavel Stěhule<br />
<br />
== Oracle ==<br />
<br />
=== Utilities, tools, scripts etc. ===<br />
<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines to PostgreSQL and back<br />
* [https://dbconvert.com/oracle/postgresql/ Database migration for Oracle and PostgreSQL] Software for cross - database conversion and sync between Oracle and PostgreSQL.<br />
* [http://ora2pg.darold.net/ Ora2Pg - Oracle to PostgreSQL database schema converter] by Gilles Darold<br />
* [http://oracle-fdw.projects.pgfoundry.org/ PostgreSQL Foreign Data Wrapper for Oracle (oracle_fdw)] - an [[Fdw|FDW]] providing support to access Oracle databases from within PostgreSQL<br />
* [http://orafce.projects.pgfoundry.org/ Orafce] - implements common Oracle functions in PostgreSQL for compatibility<br />
* [http://www.convert-in.com/ora2pgs.htm Oracle-to-PostgreSQL] - a program to migrate Oracle databases to PostgreSQL server. The program has high performance due to direct connection to data source and destination databases (it does not use ODBC or any other middleware software). Command line support allows to script, automate and schedule the conversion process.<br />
<br />
=== Documentation, articles, presentations etc. ===<br />
<br />
==== PostgreSQL documentation ====<br />
* [http://www.postgresql.org/docs/current/interactive/plpgsql-porting.html Porting from Oracle PL/SQL]<br />
<br />
==== Articles and presentations (in reverse chronological order) ====<br />
<br />
* [http://tapoueh.org/confs/2013/10/29-pgconfeu-2013 From MySQL to PostgreSQL with pgloader] (English) talk at PostgreSQL Conference Europe, Dimitri Fontaine, 2013-10-29<br />
* [http://wiki.cenatic.es/wikiesp/index.php/Procedimiento_de_An%C3%A1lisis_y_Planificaci%C3%B3n_de_Migraci%C3%B3n_de_Oracle_10g_R2_a_PostgreSQL_9.1.3 Analysis and Planning Process for Migration from Oracle 10g R2 to PostgreSQL 9.1.3] (Spanish) by the Spanish National Competence Center for the Application of Opensource Technologies [http://www.cenatic.es (CENATIC)] (2012-06)<br />
* [http://www.bostongis.com/PrinterFriendly.aspx?content_name=sqlserver2008r2_oracle11gr2_postgis15_compare Compare SQL Server 2008 R2, Oracle 11G R2, PostgreSQL/PostGIS 1.5 Spatial Features] BostonGIS (2010-06-01)<br />
* [http://wiki.kandalaya.org/cgi-bin/twiki/view/Main/MigratingOracleToPostgreSQL Migrating Oracle Databases to PostgreSQL using Xen on Debian GNU/Linux] by Raj Mathur (Kandalaya) (2008-12-26)<br />
* [http://www.pgcon.org/2008/schedule/track/Tutorial/62.en.html Porting Oracle Applications to PostgreSQL] by Peter Eisentraut (PGCon 2008)<br />
* [[PostgreSQL for Oracle DBAs]] by Richard Stephan (2007-06-01)<br />
* [[:Image:Pg 8.1 J2EE v1.0.pdf|Porting JDBC applications from Oracle to PostgreSQL]] by Chris Drawater (2006-03-24)<br />
** [http://www.holindis.co.uk/ Related documents for J2EE, Tomcat, and Oracle migrations] by Chris Drawater (2007-01-15)<br />
* [[Interview with Mark Stosberg (From Oracle/tcl to PostgreSQL/Perl)]] by Robert Treat (2004-10-24)<br />
* [[Oracle to Postgres Conversion]] by James Shannon, Ben Adida, and Don Baccus<br />
<br />
== Progress RDBMS ==<br />
<br />
* [http://knol.google.com/k/tomasz-judycki/convert-progress-rdbms-to-postgresql/2rifd79ifooa9/4# Convert Progress RDBMS to PostgreSQL] by Tomasz Judycki<br />
<br />
== Converting PostgreSQL Databases to other Databases ==<br />
<br />
* [http://docman.sourceforge.net/home_html/projects/sql/pgsql2interbase PostgreSQL to InterBase dump file converter] by Dobrica Pavlinusic<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from PostgreSQL to more than 30 database engines<br />
* [https://github.com/OmniDB/OmniDB OmniDB] Open source web tool for database management and conversion. Currently supports PostgreSQL, Oracle, MariaDB, MySQL, Firebird, SQLite, MS Access, MS SQL Server, MS SQL Compact. OmniDB converts from any supported RDBMS to PostgreSQL and back.<br />
<br />
[[Category:Install]]<br />
[[Category:General articles and guides]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Converting_from_other_Databases_to_PostgreSQL&diff=28773Converting from other Databases to PostgreSQL2016-12-10T18:15:51Z<p>Rjuju: add db2topg migration tool</p>
<hr />
<div>== Non-specific ==<br />
<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines to PostgreSQL and back<br />
* [https://github.com/dimitri/pgloader pgloader] knows how to load data from MySQL, SQLite, MS SQL Server, dBase files, CSV files and fixed-width data files, and more. Released under The PostgreSQL Licence.<br />
* [https://dbconvert.com/postgresql/ DBConvert PostgreSQL database migration and sync software] Database conversion and synchronization between PostgreSQL/ Amazon RDS, MySQL, MS SQL Server, SQL Azure, Oracle, MS Access.<br />
* [http://www.easyfrom.net/ Converting data between PostgreSQL and others database formats] ESF Database Migration Toolkit enables you to transfer data across various databases, supporting PostgreSQL, MySQL, Oracle, SQL Server, IBM DB2, Informix, Microsoft Access, Microsoft Excel, dBase, Foxpro, Firbird, SQLite etc. - by Martin Williams<br />
* [http://troels.arvin.dk/db/rdbms/ Comparison of different SQL implementations] by Troels Arvin (covers PG 8.4 and MySQL 5.0)<br />
* [[Transactional DDL in PostgreSQL: A Competitive Analysis]] by Greg Smith<br />
* [[Migrating from one database to another with Pentaho ETL]] by Nicola Benaglia<br />
* [http://www.vive.net/products/datapro.htm dataPro] Conversion tool for PostgreSQL, SQLite, MySQL, Oracle, SQL Server and Microsoft Access. Transfer database objects between different databases and servers, convert tables schema and migrate data from one database type to another.<br />
* [http://www.datanamic.com/datadiff-crossdb/ DataDiff CrossDB] is a Windows GUI utility to compare and synchronize/transfer data from PostgreSQL to/from Oracle, MSSQL, MS Access or MySQL databases.<br />
* [http://www.sqlmaestro.com/products/postgresql/datawizard/ PostgreSQL Data Wizard] is a Windows GUI utility to transfer both schema and data from any ADO-compatible source (like MS Access, MySQL, SQL Server, Oracle, etc) to PostgreSQL.<br />
* [https://metacpan.org/module/SQL::Translator SQL::Translator] is a Perl module for translating table definitions between different software.<br />
* [[Foreign_data_wrappers|Foreign data wrappers]] may be useful for exporting data from other databases<br />
* [http://www.convert-in.com/pgskit.htm Postgres Migration Toolkit] Software pack to convert Oracle, MySQL, SQL Server and FoxPro to PostgreSQL, and vice versa.<br />
* [https://github.com/ei-grad/sqlacrossover sqlacrossover] SQLAlchemy-based cross-database migration tool<br />
* [https://skyvia.com Skyvia] Web service for cloud data integration for PostgreSQL with Salesforce, Dynamics CRM, SugarCRM, Zoho CRM, QuickBooks, FreshBooks, ExactTarget, MailChimp, Bigcommerce, MySQL, SQL Server, SQL Azure, Amazon RDS.<br />
* [https://github.com/OmniDB/OmniDB OmniDB] Open source web tool for database management and conversion. Currently supports PostgreSQL, Oracle, MariaDB, MySQL, Firebird, SQLite, MS Access, MS SQL Server, MS SQL Compact. OmniDB converts from any supported RDBMS to PostgreSQL and back.<br />
<br />
== DBase II, III, IV+ / DBF Format ==<br />
<br />
* [http://www.convert-in.com/dbf2pgs.htm FoxPro-to-PostgreSQL] a program to migrate FoxPro databases to PostgreSQL server. The program does not use ODBC or any other middleware software. Command line support allows to script, automate and schedule the conversion process.<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from DBF (dBase, Clipper, Visual FoxPro and other DBF variants) to PostgreSQL and back<br />
* [https://github.com/dimitri/pgloader pgloader] by Dimitri Fontaine<br />
* [http://www.tv.com.pl/stepbystep/dbasepsql/ Convert .dbf files into PostgreSQL] by Tomasz Judycki<br />
* [[:Image:26.zip|A Visual Basic utility to convert Dbase III, IV, and V]] by Dennis Bazan<br />
* [[Porting data from dBASE IV to PostgreSQL]] by Vijay Deval (2002-09-07)<br />
* [https://github.com/kstrauser/pgdbf PgDBF : Simplified and optimized replacement for XBaseToPg] by Kirk Strauser<br />
<br />
== FileMaker Pro ==<br />
<br />
* [[Porting from FileMaker Pro to PostgreSQL]] by Michelle Murrain (24th August 2001)<br />
<br />
== IBM DB2 ==<br />
<br />
* [[:Image:DB2UDB-to-PG.pdf|Migrating from DB2 to PostgreSQL]]<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including IBM DB2, to PostgreSQL and back<br />
* [[https://github.com/dalibo/db2topg/ db2topg]] Migration tool to convert a DB2 UDB Database into a PostgreSQL database<br />
<br />
== Interbase ==<br />
<br />
* [https://github.com/kstrauser/dbreplicate DBReplicate - Simplify Interbase->PostgreSQL conversions] by Kirk Strauser<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including Interbase, to PostgreSQL and back<br />
<br />
== Microsoft Access ==<br />
<br />
* [http://www.postgresonline.com/journal/index.php?url=archives/346-Querying-MS-Access-and-other-ODBC-data-sources-with-OGR_FDW.html Linking to MS Access tables with ogr_fdw] If your PostgreSQL is on windows, you can use ogr_fdw foreign data wrapper, packaged with PostGIS 2.2+ bundle for windows via application stackbuilder. With PostgresQL 9.5, you can use IMPORT FOREIGN SCHEMA to link in all the MS access tables and then cherry pick how you want to restructure.<br />
* [http://docman.sourceforge.net/home_html/projects/sql/exportSQL3.txt exportSQL] - a Microsoft Access module which exports Access Database into MySQL, mSQL and PostgreSQL by Dobrica Pavlinusic. Based on the work of Pedro Freire<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including Access, to PostgreSQL and back<br />
* [http://mdbtools.sourceforge.net/ MDB Tools] by Brian Bruns<br />
** A quick way to dump all tables as tsv or csv files<br />
for TT in $(mdb-tables file.mdb); do<br />
mdb-export -Q -d '\t' -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.tsv"<br />
done<br />
<br />
for TT in $(mdb-tables file.mdb); do<br />
mdb-export -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.csv"<br />
done<br />
<br />
If the tablenames have embedded spaces...<br />
<br />
mdb-tables -1 file.mdb| while read TT<br />
do<br />
mdb-export -D '%Y-%m-%d %H:%M:%S' file.mdb "$TT" > "${TT}.csv"<br />
done<br />
<br />
A shell script that may be useful for converting entire databases:<br />
<br />
#!/bin/sh -e<br />
<br />
mdbfn=$1<br />
schemafn=$2<br />
fkfn=$3<br />
datafn=$4<br />
schema=$5<br />
<br />
tf=$(tempfile)<br />
<br />
pre=""<br />
[ -n "${schema}" ] && pre="\"${schema}\"."<br />
<br />
mdb-schema "${mdbfn}" postgres > "${tf}"<br />
<br />
# Schema file<br />
echo "BEGIN;\n" > "${schemafn}"<br />
<br />
sp=""<br />
[ -n "${schema}" ] && echo "CREATE SCHEMA \"${schema}\";\n" >> "${schemafn}"<br />
[ -n "${schema}" ] && sp="SET search_path = \"${schema}\", pg_catalog;\n" <br />
<br />
echo ${sp} >> "${schemafn}"<br />
<br />
awk '($0 !~ /^ALTER TABLE.*FOREIGN KEY.*REFERENCES/) {print;}' "${tf}" >> "${schemafn}"<br />
<br />
echo "\nEND;" >> "${schemafn}"<br />
<br />
# Foreign keys file<br />
echo "BEGIN;\n" > "${fkfn}"<br />
echo ${sp} >> "${fkfn}"<br />
<br />
awk '($0 ~ /^ALTER TABLE.*FOREIGN KEY.*REFERENCES/) {print;}' "${tf}" >> "${fkfn}"<br />
<br />
echo "\nEND;" >> "${fkfn}"<br />
<br />
# Data file<br />
echo "BEGIN;\n" > "${datafn}"<br />
echo "SET CONSTRAINTS ALL DEFERRED;\n" >> "${datafn}"<br />
<br />
mdb-tables -1 "${mdbfn}" | while read TT<br />
do<br />
mdb-export -Q -d '\t' -D '%Y-%m-%d %H:%M:%S' "${mdbfn}" "$TT" > "${tf}"<br />
<br />
awk -v pre="${pre}" -v TT="${TT}" \<br />
'(NR==1) {gsub(/\t/,"\",\""); print "COPY " pre "\"" TT "\"(\"" $0 "\") FROM stdin;";}' "${tf}" >> "${datafn}"<br />
awk '(NR>1) {gsub(/\t\t/,"\t\\N\t"); gsub(/\t$/,"\t\\N"); gsub(/\t\t/,"\t\\N\t"); print;}' "${tf}" >> "${datafn}"<br />
<br />
echo "\\.\n" >> "${datafn}"<br />
done<br />
<br />
echo "END;" >> "${datafn}"<br />
<br />
rm -f "${tf}"<br />
<br />
If this script is saved to the file access2psql.sh and made executable, then it would be used as follows:<br />
<br />
access2psql.sh file.mdb schema.sql foreignkeys.sql data.sql pg_schema_name<br />
psql -f schema.sql pg_db_name<br />
psql -f data.sql pg_db_name<br />
psql -f foreignkeys.sql pg_db_name<br />
<br />
This script won't work properly if there are tab characters in text columns, though the call to mdb-export could be modified to export INSERT statements to fix this. Also, mdb-schema has trouble representing multi-column foreign keys, so foreignkeys.sql may need some manual editing.<br />
<br />
* [http://convertdb.com/access/postgresql/ Export Microsoft Access to PostgreSQL. Cross database synchronization]<br />
* [http://www.olschimke.eu/2012/08/07/importing-microsoft-access-mdb-into-postgresql-on-linux-postgres/ Importing Microsoft Access MDB into PostgreSQL on Linux] by Michael Olschimke<br />
* [[Microsoft Access to PostgreSQL Conversion]] by Jon Hutchings (2001-07-20)<br />
* [http://pgfoundry.org/projects/access2pgsql/ access2pgsql tool] by Mariano Reingart<br />
* [http://www.postgresonline.com/journal/archives/24-Using-MS-Access-with-PostgreSQL.html Using MS Access as front end and PostgreSQL as back-end database] by PostgresOnLine Journal - 2008-01-31<br />
<br />
== Microsoft SQL Server ==<br />
<br />
* [http://www.convert-in.com/mss2pgs.htm MSSQL-to-PostgreSQL] is a migration utility to convert SQL Server or SQL Azure databases to PostgreSQL. Option to filter data using SELECT-queries, synchronization mode, command line support.<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines, including SQL Server, to PostgreSQL and back<br />
* [[http://dalibo.github.io/sqlserver2pgsql/ sqlserver2pgsql]] Migration tool to convert a Microsoft SQL Server Database into a PostgreSQL database<br />
* [https://github.com/dimitri/pgloader pgloader] by Dimitri Fontaine<br />
* [https://dbconvert.com/mssql/postgresql/ DBConvert for PostgreSQL and MS SQL Server] Conversion and Sync software solutions for cross database migration between PostgreSQL/ Amazon RDS and MS SQL Server/ SQL Azure<br />
* [http://sqlblog.com/blogs/alexander_kuznetsov/archive/tags/PostgreSql/default.aspx Learning PostgreSQL (Alexander Kuznetsov)] - series of blog articles for SQL Server users (2013-10 ~ 2013-11)<br />
* [http://www.postgresonline.com/journal/archives/219-SQL-Server-to-PostgreSQL-Converting-table-structure.html Converting SQL Server Table Structure to PostgreSQL] Leo Hsu and Regina Obe (2011-09-03)<br />
* [[:Image:5.pdf|Conversion of Microsoft SQL/ASP applications to PostgreSQL]] by Ethan Townsend (2005-06-23)<br />
* [[Converting your data from MS SQL Server 7 to PostgreSQL 7.1.x]] by Ryan C. Bonham (2002-01-05)<br />
* [[Microsoft SQL Server to PostgreSQL Migration by Ian Harding]] (2001-09-17)<br />
* [http://www.bostongis.com/PrinterFriendly.aspx?content_name=sqlserver2008r2_oracle11gr2_postgis15_compare Compare SQL Server 2008 R2, Oracle 11G R2, PostgreSQL/PostGIS 1.5 Spatial Features] BostonGIS (2010-06-01) <br />
* [http://www.postgresonline.com/journal/index.php?/archives/51-Cross-Compare-of-SQL-Server,-MySQL,-and-PostgreSQL.html Cross Compare of SQL Server 2005, MySQL 5, and PostgreSQL 8.3] by Leo Hsu and Regina Obe (2008-05-13)<br />
* [http://www.postgresonline.com/journal/index.php?/archives/130-Cross-Compare-of-PostgreSQL-8.4,-SQL-Server-2008,-MySQL-5.1.html Cross Compare of SQL Server 2008, MySQL 5.1, and PostgreSQL 8.4] by Leo Hsu and Regina Obe (2009-08-15)<br />
* [http://www.postgresonline.com/journal/archives/179-Universal-Unique-Identifiers-PostgreSQL-SQL-Server-Compare.html Universal Unique Identifiers PostgreSQL SQL Server Compare] Demonstrates how to implement SQL Server NEWID() GUID uniqueidentifier type in PostgreSQL by Leo Hsu and Regina Obe (2010-10-07)<br />
<br />
== MySQL ==<br />
<br />
<br />
=== Scripts, programs ===<br />
<br />
==== 2015 ====<br />
* [http://www.convert-in.com/sql2pgs.htm MySQL-to-PostgreSQL] is a program to migrate MySQL databases to PostgreSQL server. Option to filter data using SELECT-queries, synchronization mode, command line support.<br />
* [https://github.com/AnatolyUss/FromMySqlToPostgreSql FromMySqlToPostgreSql migration tool by Anatoly Khaytovich], provides an accurate migration of table data, indices, PKs, FKs... Makes an extensive use of PostgreSQL COPY protocol.<br />
<br />
==== 2014 ====<br />
<br />
* [https://github.com/mihailShumilov/mysql2postgresql MySQL/PostgreSQL Converter from Mihail Shumilov], written on PHP. Supported big DB<br />
<br />
==== 2013 ====<br />
* [https://github.com/lanyrd/mysql-postgresql-converter MySQL/PostgreSQL Converter from Lanyrd], based on [http://lanyrd.com/blog/2012/lanyrds-big-move/ work detailed here]<br />
* [https://github.com/dimitri/pgloader pgloader], from Dimitri Fontaine, based on [http://tapoueh.org/blog/2013/01/28-pgloader-future work detailed here] and [http://tapoueh.org/blog/2013/11/12-migrating-sakila Migrating Sakila from MySQL to PostgreSQL]<br />
* [https://github.com/philipsoutham/py-mysql2pgsql py-mysql2pgsql], [https://github.com/mozilla/airmozilla/pull/61/files#diff-30 example use case here]<br />
* [https://github.com/bdigital/my_etl MyETL], Tool that works with the [https://github.com/bdigital/mysql_fdw MySQL FDW]<br />
<br />
==== Previously ====<br />
* [http://www.enterprisedb.com/products/download.do Free MySQL to Postgres Migration Wizard by EnterpriseDB v1.1] by EnterpriseDB Corporation. Available without registration via PostgreSQL installer wizard.<br />
* [http://www.pgsql.com/download/ Conversion tool for migrating from MySQL to PostgreSQL] by PostgreSQL Inc.<br />
* [http://en.wikibooks.org/wiki/Converting_MySQL_to_PostgreSQL Converting MySQL to PostgreSQL] - from en.wikibooks.org<br />
* [http://convertdb.com/mysql/postgresql/ Migrate MySQL to PostgreSQL.] - Cross database bidirectional synchronization<br />
* [http://www.sourcefiles.org/Databases/Utilities/Conversion/my2pg.pl my2pg.pl] - A Perl script used to convert a MySQL database dump to PostgreSQL-compatible format, by Maxim Rudensky and Valentine Danilchuk<br />
* [http://pgfoundry.org/projects/mysql2pgsql/ mysql2pgsql] - A Perl script used to convert MySQL databases dump to a PostgreSQL-compatible format<br />
* [http://mp2p.mikekohn.net/ MySQL PHP to PostgreSQL] by Michael Kohn<br />
* [http://www.gab.lc/script_php_my2pg.php PHP_my2pg] PHP script by Gabriel Bordeaux<br />
* [http://tryolabs.com/Blog/2012/02/10/django-migrating-mysql-postgresql/ Migrating from MySQL to Postgresql with Django] blogpost by alejandro<br />
* [http://github.com/maxlapshin/mysql2postgres Migrates MySQL to PostgreSQL] Ruby script by Max Lapshin<br />
# To install mysql2psql (under ubuntu 11.10): No need to get from github, just:<br />
sudo apt-get install ruby gems libmysqlclient-dev libpq-dev<br />
gem install mysql pg mysql2psql<br />
# To get info about the mysql socket:<br />
netstat -l | grep mysql<br />
mysql2psql # creates a .yml templae<br />
vi mysql2psql.yml # edit the template<br />
mysql2psql # connects to mysql database and write into postgres database<br />
<br />
* [http://pypi.python.org/pypi/py-mysql2pgsql py-mysql2pgsql] Python script to convert MySQL database to PostgreSQL compatible file or export directly to a PostgreSQL server<br />
* [https://bitbucket.org/iceone/mysql2pgsql/ my2pg.py] MySQL to PostgreSQL database conversion (not the same as above)<br />
* [http://phpseclib.sourceforge.net/sql/ SQL Data Definition Language (DDL) Conversion to MySQL, PostgreSQL and MS-SQL] by phpBB Group<br />
<br />
=== Documentation ===<br />
<br />
* [http://www.sitepoint.com/article/site-mysql-postgresql-1 Backend Database Switcheroo Howto Part I] by Nathan Matias<br />
* [http://www.sitepoint.com/article/site-mysql-postgresql-2 Backend Database Switcheroo Howto Part II] by Nathan Matias<br />
* [http://www.xach.com/aolserver/mysql-to-postgresql.html How-To: Migrating from MySQL to PostgreSQL] by Zach Beane<br />
* [[Things to find out about when moving from MySQL to PostgreSQL]] by Joel Burton (8th April 2001)<br />
* [[Why PostgreSQL Instead of MySQL: Comparing Reliability and Speed in 2007]] by Greg Smith<br />
* [http://www.postgresonline.com/journal/index.php?/archives/51-Cross-Compare-of-SQL-Server,-MySQL,-and-PostgreSQL.html Cross Compare of SQL Server, MySQL, and PostgreSQL] by Leo Hsu and Regina Obe (2008-05-13)<br />
* [[How to make a proper migration from MySQL to PostgreSQL]]<br />
* [http://andreas.scherbaum.la/writings/migrate_web_platforms.pdf Migrating a community platform from Mysql to PostgreSQL] by Andreas Scherbaum (Prato, 2007)<br />
* [http://okbob.blogspot.com/2009/08/mysql-functions-for-postgresql.html MySQL functions for PostgreSQL] by Pavel Stěhule<br />
<br />
== Oracle ==<br />
<br />
=== Utilities, tools, scripts etc. ===<br />
<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from more than 30 database engines to PostgreSQL and back<br />
* [https://dbconvert.com/oracle/postgresql/ Database migration for Oracle and PostgreSQL] Software for cross - database conversion and sync between Oracle and PostgreSQL.<br />
* [http://ora2pg.darold.net/ Ora2Pg - Oracle to PostgreSQL database schema converter] by Gilles Darold<br />
* [http://oracle-fdw.projects.pgfoundry.org/ PostgreSQL Foreign Data Wrapper for Oracle (oracle_fdw)] - an [[Fdw|FDW]] providing support to access Oracle databases from within PostgreSQL<br />
* [http://orafce.projects.pgfoundry.org/ Orafce] - implements common Oracle functions in PostgreSQL for compatibility<br />
* [http://www.convert-in.com/ora2pgs.htm Oracle-to-PostgreSQL] - a program to migrate Oracle databases to PostgreSQL server. The program has high performance due to direct connection to data source and destination databases (it does not use ODBC or any other middleware software). Command line support allows to script, automate and schedule the conversion process.<br />
<br />
=== Documentation, articles, presentations etc. ===<br />
<br />
==== PostgreSQL documentation ====<br />
* [http://www.postgresql.org/docs/current/interactive/plpgsql-porting.html Porting from Oracle PL/SQL]<br />
<br />
==== Articles and presentations (in reverse chronological order) ====<br />
<br />
* [http://tapoueh.org/confs/2013/10/29-pgconfeu-2013 From MySQL to PostgreSQL with pgloader] (English) talk at PostgreSQL Conference Europe, Dimitri Fontaine, 2013-10-29<br />
* [http://wiki.cenatic.es/wikiesp/index.php/Procedimiento_de_An%C3%A1lisis_y_Planificaci%C3%B3n_de_Migraci%C3%B3n_de_Oracle_10g_R2_a_PostgreSQL_9.1.3 Analysis and Planning Process for Migration from Oracle 10g R2 to PostgreSQL 9.1.3] (Spanish) by the Spanish National Competence Center for the Application of Opensource Technologies [http://www.cenatic.es (CENATIC)] (2012-06)<br />
* [http://www.bostongis.com/PrinterFriendly.aspx?content_name=sqlserver2008r2_oracle11gr2_postgis15_compare Compare SQL Server 2008 R2, Oracle 11G R2, PostgreSQL/PostGIS 1.5 Spatial Features] BostonGIS (2010-06-01)<br />
* [http://wiki.kandalaya.org/cgi-bin/twiki/view/Main/MigratingOracleToPostgreSQL Migrating Oracle Databases to PostgreSQL using Xen on Debian GNU/Linux] by Raj Mathur (Kandalaya) (2008-12-26)<br />
* [http://www.pgcon.org/2008/schedule/track/Tutorial/62.en.html Porting Oracle Applications to PostgreSQL] by Peter Eisentraut (PGCon 2008)<br />
* [[PostgreSQL for Oracle DBAs]] by Richard Stephan (2007-06-01)<br />
* [[:Image:Pg 8.1 J2EE v1.0.pdf|Porting JDBC applications from Oracle to PostgreSQL]] by Chris Drawater (2006-03-24)<br />
** [http://www.holindis.co.uk/ Related documents for J2EE, Tomcat, and Oracle migrations] by Chris Drawater (2007-01-15)<br />
* [[Interview with Mark Stosberg (From Oracle/tcl to PostgreSQL/Perl)]] by Robert Treat (2004-10-24)<br />
* [[Oracle to Postgres Conversion]] by James Shannon, Ben Adida, and Don Baccus<br />
<br />
== Progress RDBMS ==<br />
<br />
* [http://knol.google.com/k/tomasz-judycki/convert-progress-rdbms-to-postgresql/2rifd79ifooa9/4# Convert Progress RDBMS to PostgreSQL] by Tomasz Judycki<br />
<br />
== Converting PostgreSQL Databases to other Databases ==<br />
<br />
* [http://docman.sourceforge.net/home_html/projects/sql/pgsql2interbase PostgreSQL to InterBase dump file converter] by Dobrica Pavlinusic<br />
* [https://www.spectralcore.com/fullconvert Full Convert] Database conversion from PostgreSQL to more than 30 database engines<br />
* [https://github.com/OmniDB/OmniDB OmniDB] Open source web tool for database management and conversion. Currently supports PostgreSQL, Oracle, MariaDB, MySQL, Firebird, SQLite, MS Access, MS SQL Server, MS SQL Compact. OmniDB converts from any supported RDBMS to PostgreSQL and back.<br />
<br />
[[Category:Install]]<br />
[[Category:General articles and guides]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_Conference_Europe_Talks_2016&diff=28568PostgreSQL Conference Europe Talks 20162016-11-07T20:06:25Z<p>Rjuju: add slides for the talk "BRIN indexes on geospatial databases"</p>
<hr />
<div>pgconf.eu 2016 took place in Tallinn, Estonia, from 2016-11-01 to 2016-11-04.<br />
<br />
== Tuesday (Trainings) ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Trainer||Title (Link to slides)<br />
|+<br />
| 09:00 - 17:00 || Alfa 2 || Petr Jelinek, Simon Riggs || PostgreSQL Replication & Upgrades<br />
|}<br />
<br />
== Wednesday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 11:10 - 12:00 || Alfa 2 || David Steele || [https://github.com/dwsteele/conference/blob/release/PostgresPatchReview-PGConfEU-2016/slides/slides.pdf Reviewing PostgreSQL Patches for Fun and Profit]<br />
|-<br />
| 14:00 - 14:50 || Omega || Alexander Korotkov || [http://www.slideshare.net/AlexanderKorotkov/the-future-is-csn The future is CSN]<br />
|-<br />
| 15:00 - 15:50 || Alfa 2 || Andreas Seltenreich || [https://korte.credativ.com/~ase/sqlsmith-talk.pdf Hunting PostgreSQL bugs with SQLsmith]<br />
|-<br />
| 16:20 - 17:10 || Alfa 2 || Jehan-Guillaume (ioguix) de Rorthais || [http://www.dalibo.org/_media/2016-pgconfeu-paf.html.gz PAF: auto failover and more!]<br />
|-<br />
| 17:20 - 18:10 || Alfa 2 || Emre Hasegeli || [http://www.slideshare.net/EmreHasegeli/managing-thousands-of-databases Managing Thousands of Databases]<br />
|}<br />
<br />
== Thursday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|-<br />
| 11:50 - 12:40 || Alfa 2 || Giuseppe Broccolo & Julien Rouhaud || [http://www.dalibo.org/_media/gbroccolo_jrouhaud_pgconfeu2016_brin4postgis.pdf BRIN indexes on geospatial databases]<br />
|+<br />
| 15:40 - 16:30 || Alfa 2 || Cédric Villemain || [[Media:2ndQUadrant Datastore xa.pdf|Transactions across multiple datastores]]<br />
|}<br />
<br />
== Friday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 09:30 - 10:20 || Omega || Oleg Bartunov, Artur Zakirov || [http://www.slideshare.net/ArthurZakirov1/better-full-text-search-in-postgresql Better Full Text Search in PostgreSQL]<br />
|-<br />
| 10:50 - 11:40 || Alfa 2 || Vik Fearing || [[:File:Lateral_tallinn.pdf|How Did We Live Without LATERAL?]]<br />
|-<br />
| 11:50 - 12:40 || Alfa 1 || David Steele || [https://github.com/dwsteele/conference/blob/release/AuditLogging-PGConfEU-2016/slides.pdf Audit Logging for PostgreSQL]<br />
|-<br />
| 11:50 - 12:40 || Alfa 2 || Alexey Lesovsky || [http://www.slideshare.net/alexeylesovsky/manage-postgresql-with-pgcenter Managing PostgreSQL with pgCenter]<br />
|-<br />
| 13:40 - 14:30 || Omega || Cédric Villemain || [[Media:2ndQuadrant Stats talk.pdf|A close look at stats: what you can do with them !]]<br />
|}</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_Conference_Europe_Talks_2016&diff=28567PostgreSQL Conference Europe Talks 20162016-11-07T19:51:51Z<p>Rjuju: pgcenter talk was on friday</p>
<hr />
<div>pgconf.eu 2016 took place in Tallinn, Estonia, from 2016-11-01 to 2016-11-04.<br />
<br />
== Tuesday (Trainings) ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Trainer||Title (Link to slides)<br />
|+<br />
| 09:00 - 17:00 || Alfa 2 || Petr Jelinek, Simon Riggs || PostgreSQL Replication & Upgrades<br />
|}<br />
<br />
== Wednesday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 11:10 - 12:00 || Alfa 2 || David Steele || [https://github.com/dwsteele/conference/blob/release/PostgresPatchReview-PGConfEU-2016/slides/slides.pdf Reviewing PostgreSQL Patches for Fun and Profit]<br />
|-<br />
| 14:00 - 14:50 || Omega || Alexander Korotkov || [http://www.slideshare.net/AlexanderKorotkov/the-future-is-csn The future is CSN]<br />
|-<br />
| 15:00 - 15:50 || Alfa 2 || Andreas Seltenreich || [https://korte.credativ.com/~ase/sqlsmith-talk.pdf Hunting PostgreSQL bugs with SQLsmith]<br />
|-<br />
| 16:20 - 17:10 || Alfa 2 || Jehan-Guillaume (ioguix) de Rorthais || [http://www.dalibo.org/_media/2016-pgconfeu-paf.html.gz PAF: auto failover and more!]<br />
|-<br />
| 17:20 - 18:10 || Alfa 2 || Emre Hasegeli || [http://www.slideshare.net/EmreHasegeli/managing-thousands-of-databases Managing Thousands of Databases]<br />
|}<br />
<br />
== Thursday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 15:40 - 16:30 || Alfa 2 || Cédric Villemain || [[Media:2ndQUadrant Datastore xa.pdf|Transactions across multiple datastores]]<br />
|}<br />
<br />
== Friday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 09:30 - 10:20 || Omega || Oleg Bartunov, Artur Zakirov || [http://www.slideshare.net/ArthurZakirov1/better-full-text-search-in-postgresql Better Full Text Search in PostgreSQL]<br />
|-<br />
| 10:50 - 11:40 || Alfa 2 || Vik Fearing || [[:File:Lateral_tallinn.pdf|How Did We Live Without LATERAL?]]<br />
|-<br />
| 11:50 - 12:40 || Alfa 1 || David Steele || [https://github.com/dwsteele/conference/blob/release/AuditLogging-PGConfEU-2016/slides.pdf Audit Logging for PostgreSQL]<br />
|-<br />
| 11:50 - 12:40 || Alfa 2 || Alexey Lesovsky || [http://www.slideshare.net/alexeylesovsky/manage-postgresql-with-pgcenter Managing PostgreSQL with pgCenter]<br />
|-<br />
| 13:40 - 14:30 || Omega || Cédric Villemain || [[Media:2ndQuadrant Stats talk.pdf|A close look at stats: what you can do with them !]]<br />
|}</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Monitoring&diff=27206Monitoring2016-02-29T21:11:13Z<p>Rjuju: </p>
<hr />
<div>== PostgreSQL builtin & contrib ==<br />
<br />
=== Statistics collector ===<br />
<br />
PostgreSQL collects lots of data on its own and offers it via the <tt>pg_stat(io)_</tt> system views<br />
<br />
* Official documentation on the [http://www.postgresql.org/docs/current/static/monitoring-stats.html Statistics Collector]<br />
* [http://www.varlena.com/GeneralBits/107.php Interpreting pg_stat Views]<br />
<br />
=== contrib extensions ===<br />
<br />
The following extensions offer access to Postgres internals which may be of interest or collect additional information. Most of them are shipped with Postgres (the <tt>-contrib</tt> packages may need to be installed) and can be activated via the [http://www.postgresql.org/docs/current/static/sql-createextension.html extension interface].<br />
<br />
==== pg_stat_statements ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstatstatements.html pg_stat_statements]</tt> tracks all queries that are executed on the server and records average runtime per query "class" among other parameters.<br />
<br />
==== pg_stat_plans ====<br />
<br />
<tt>[https://github.com/2ndQuadrant/pg_stat_plans pg_stat_plans]</tt> extends on pg_stat_statements and records query plans for all executed quries. This is very helpful when you're experiencing performance regressions due to inefficient query plans due to changed parameters or table sizes.<br />
<br />
==== pgstattuple ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstattuple.html pgstattuple]</tt> can generate statistics for tables and indexes, showing how much space in each table & index is consumed by live tuples, deleted tuples as well as how much unused space is available in each relation.<br />
<br />
==== pg_buffercache ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgbuffercache.html pg_buffercache]</tt> gives you introspection into Postgres' [http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-SHARED-BUFFERS shared buffers], showing how many pages of which relations are currently held in the cache.<br />
<br />
== External projects ==<br />
<br />
=== CLI tools ===<br />
<br />
==== pg_view ====<br />
<br />
<tt>[http://tech.zalando.com/getting-a-quick-view-of-your-postgresql-stats/ pg_view]</tt> is a Python-based tool to quickly get information about running databases and resources used by them as well as correlate running queries and why they might be slow.<br />
<br />
==== pg_activity ====<br />
<br />
<tt>[https://github.com/julmon/pg_activity pg_activity]</tt> is a htop like application for PostgreSQL server activity monitoring, written in Python.<br />
<br />
==== pgstats ====<br />
<br />
<tt>[https://github.com/gleu/pgstats pgstats]</tt> is a command line tool written in C which can sample various PostgreSQL informations. It also provides a tool to generate CSV files to graph the pgstats metrics.<br />
=== Checkers === <br />
<br />
==== check_postgres ====<br />
<br />
<tt>[http://bucardo.org/wiki/Check_postgres check_postgres]</tt> is a command line tool which is designed to be run from software like Icinga, MRTG or as a standalone tool. It can monitor many aspects of the database and trigger warnings when thresholds are violated.<br />
<br />
=== Interfaces & collectors ===<br />
<br />
These tools either offer an interface to PostgreSQL monitoring-relevant data or can aggregate and prepare them for other systems.<br />
<br />
==== pgsnmpd ====<br />
<br />
<tt>[http://pgsnmpd.projects.postgresql.org/ pgsnmpd]</tt> can run as a standalone SNMP server and implements the [http://www.faqs.org/rfcs/rfc1697.html RFC 1697 MIB] which is generic RDBMS [http://en.wikipedia.org/wiki/Management_information_base MIB]<br />
<br />
This is useful for network management systems which are limited to SNMP protocol.<br />
<br />
==== pganalyze-collector ====<br />
<br />
<tt>[https://github.com/pganalyze/pganalyze-collector pganalyze-collector]</tt> is a tool which collects <tt>pg_stat_plans</tt> query information as well as various performance-relevant database parameters and converts them into a JSON structure for easy ingestion in other systems.<br />
<br />
=== Generic monitoring solutions with plugins ===<br />
<br />
====Datadog====<br />
[http://www.datadoghq.com/ Datadog] is a commercial saas that collects postgres metrics on connections, transactions, row crud operations, locks, temp files, bgwriter, index usage, replication status, memory, disk, cpu and lets you visualize and alert on those metrics alongside your other system and application metrics.<br />
<br />
==== Munin ====<br />
<br />
PostgreSQL Plugins developed in Perl are included in the Core [http://munin-monitoring.org/ Munin] Distribution. The following plugins are included by default: <tt>postgres_bgwriter, postgres_locks_, postgres_tuples_, postgres_cache_, postgres_querylength_, postgres_users, postgres_checkpoints, postgres_scans_, postgres_xlog, postgres_connections_, postgres_size_, postgres_connections_db, postgres_transactions_</tt><br />
<br />
[http://aouyar.github.com/PyMunin/ PyMunin] includes a Multigraph Munin Plugin written in Python that implements the following graphs: <tt>pg_connections, pg_diskspace, pg_blockreads, pg_xact, pg_tup_read, pg_tup_write, pg_blockreads_detail, pg_xact_commit_detail, pg_xact_rollback_detail, pg_tup_return_detail, pg_tup_fetch_detail, pg_tup_delete_detail, pg_tup_update_detail, pg_tup_insert_detail</tt><br />
<br />
Detailed setup instructions for common Linux platforms can be found at [http://highperfpostgres.com/guides/postgresql-monitoring-with-munin/ highperfpostgres.com]<br />
<br />
==== Zabbix ====<br />
<br />
[http://pg-monz.github.io/pg_monz/index-en.html pg_monz] is a [http://www.zabbix.com/ Zabbix] monitoring template for PostgreSQL.<br />
<br />
[http://cavaliercoder.github.io/libzbxpgsql/ libzbxpgsql] is a [http://www.zabbix.com/ Zabbix] monitoring template and native agent module for PostgreSQL.<br />
<br />
==== NewRelic ====<br />
<br />
[http://newrelic.com/ NewRelic] is a commercial SaaS application monitoring solution which offers a [https://newrelic.com/plugins/enterprisedb-corporation/30 PostgreSQL plugin] maintained by EnterpriseDB.<br />
<br />
==== Cacti ====<br />
<br />
There has been work done on building a Postgres template for [http://www.cacti.net/ Cacti], Details can be found at the [[Cacti]] page.<br />
<br />
=== Postgres-centric monitoring solutions ===<br />
<br />
==== EnterpriseDB Postgres Enterprise Manager ====<br />
<br />
[http://www.enterprisedb.com/products-services-training/products/postgres-enterprise-manager Postgres Enterprise Manager] is a commercial application offered by EnterpriseDB which covers many aspects of Postgres operations & monitoring in large environments.<br />
<br />
==== pganalyze ====<br />
<br />
[https://pganalyze.com/ pganalyze] is a commercial SaaS offering which focuses on performance monitoring and automated tuning suggestions.<br />
<br />
==== pgwatch ====<br />
<br />
[http://www.cybertec.at/en/products/pgwatch-cybertec-enterprise-postgresql-monitor/ pgwatch] is a PHP web application which offers interactive graphs for relevant Postgres data.<br />
<br />
==== pg_statsinfo & pg_stats_reporter ====<br />
<br />
[http://pgstatsinfo.projects.pgfoundry.org/ pg_statsinfo] is a Postgres extension that collects lots of performance-relevant information inside the Postgres server which then can be aggregated by pg_stats_reporter instances which provide a web interface to the collected data. Both are FOSS software maintained by NTT.<br />
<br />
==== PGObserver ====<br />
<br />
[http://zalando.github.io/PGObserver/ PGObserver] is a Python & Java-based Postgres monitoring solution developed by Zalando. It was developed with a focus on stored procedure performance but extended well beyond that. <br />
<br />
==== pgCluu ====<br />
<br />
[http://pgcluu.darold.net/ pgCluu] is a Perl-based monitoring solution which uses psql and [http://en.wikipedia.org/wiki/Sar_(Unix) sar] to collect information about Postgres servers and render comprehensive performance stats.<br />
<br />
==== PoWA ==== <br />
<br />
[http://dalibo.github.io/powa/ PoWA] is a PostgreSQL Workload Analyzer that gathers performance stats and provides real-time charts and graphs to help monitor and tune your PostgreSQL servers. It relies on extensions such as pg_stat_statements, pg_qualstats, pg_stat_kcache, pg_track_settings and HypoPG, and can help you optimize you database easily.<br />
<br />
==== OPM: Open PostgreSQL Monitoring ==== <br />
<br />
[http://opm.io Open PostgreSQL Monitoring (OPM)] is a free software suite designed to help you manage your PostgreSQL servers. It's a flexible tool that will follow the activity of each instance. It can gather stats, display dashboards and send warnings when something goes wrong. The long-term goal of the project is to provide similar features to those of Oracle Grid Control or SQL Server Management Studio.<br />
<br />
==== pgAnalytics ==== <br />
<br />
[http://pganalytics.io pgAnalytics] is a comercial SaaS developed by Dextra. pgAnalytics collects snapshots from PostgreSQL statistics, log files and backups. Linux and Windows statistics are gathered to provide a better understand about whole PostgreSQL environment. Batch process are triggered to looking for errors or things that can be improved, sending alert emails. Try a [http://pganalytics.io/demo demo].<br />
<br />
[[Category:Monitoring|!]]<br />
[[Category:Performance]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Pg_session_7&diff=25982Pg session 72015-10-01T06:53:34Z<p>Rjuju: </p>
<hr />
<div>==Schedule==<br />
<br />
Schedule is available at the following website: http://www.postgresql-sessions.org/7/start<br />
<br />
== Talks: Thursday, September 24th, 2015 ==<br />
<br />
* [http://www.postgresql-sessions.org/_media/7/pg_rewind-presentation-paris.pdf pg_rewind - resynchronizing servers after failover], Heikki Linnakangas - Pivotal<br />
* [http://www.postgresql-sessions.org/_media/7/pg_session_pg-postgis-securite-interieure_last.pdf Postgresql-Postgis, les usages pour la sécurité intérieure], Vincent Laborie & Éric Pommereau<br />
* [http://www.postgresql-sessions.org/_media/7/migration_sous_linux-transparents.pdf Exemple d'une migration de PostgreSQL sous Linux d'un système de suivi de production], Nicolas Relange<br />
* [http://www.postgresql-sessions.org/_media/7/pgsession_7_postgis_tips.pdf PostGIS : trucs et astuces ], Olivier Courtin<br />
* [http://www.postgresql-sessions.org/_media/7/tempus_light.pdf présentation du projet Tempus], Olivier Courtin<br />
* [http://www.postgresql-sessions.org/_media/7/office_.net_et_postgresql.pdf http://www.postgresql-sessions.org/_media/7/office_.net_et_postgresql.pdf Office, .NET & … PostgreSQL : tiercé gagnant pour les institutions publiques ?], Jean-Marc Guazzo<br />
* [http://www.postgresql-sessions.org/_media/7/postgresql95.pdf What's new in PostgreSQL 9.5 ?], Magnus Hagander<br />
* [http://www.postgresql-sessions.org/_media/7/presentation-pgsession-24092015_v8.4.pdf Témoignage utilisateur sur la mise en œuvre de la réplication], Michel Edwell<br />
* [http://www.postgresql-sessions.org/_media/7/powa_pgsession7.pdf Optimisation de requêtes, avec PoWA / pg_qualstats / pg_stat_kcache / HypoPG], Julien Rouhaud & Ronan Dunklau - Dalibo<br />
* [http://www.postgresql-sessions.org/_media/7/pgsessions7-oracle_comment_s_en_passer.pdf Oracle, comment peut-on s'en passer ?], Gilles Darold - Dalibo<br />
* [http://www.postgresql-sessions.org/_media/7/postgresql_conf.pdf Retour d'expérience sur le remplacement d'une base comme CouchBase par PostgreSQL, et son importance croissante dans les portfolios de bases de données de cette banque], Emmanuel Remy - CASDEN<br />
* [http://www.postgresql-sessions.org/_media/7/pg_day_pres_sodexi.pdf utilisateur de PostgreSQL depuis 2009], Fabrice Noël - SoDExI<br />
<br />
== Video ==<br />
<br />
Thanks to the Mozilla Foundation, [https://www.youtube.com/playlist?list=PLdz5EN2NV_7BXtGhlWNWepg0HCJ68KXRk the talks are available here].</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Pg_session_7&diff=25963Pg session 72015-09-30T15:01:53Z<p>Rjuju: Created page with "==Schedule== Schedule is available at the following website: http://www.postgresql-sessions.org/7/start == Talks: Thursday, September 24th, 2015 == * [http://hlinnaka.iki.f..."</p>
<hr />
<div>==Schedule==<br />
<br />
Schedule is available at the following website: http://www.postgresql-sessions.org/7/start<br />
<br />
== Talks: Thursday, September 24th, 2015 ==<br />
<br />
* [http://hlinnaka.iki.fi/presentations/NordicPGDay2015-pg_rewind.pdf pg_rewind - resynchronizing servers after failover], Heikki Linnakangas - Pivotal<br />
* [http://www.postgresql-sessions.org/_media/7/pg_session_pg-postgis-securite-interieure_last.pdf Postgresql-Postgis, les usages pour la sécurité intérieure], Vincent Laborie & Éric Pommereau<br />
* [http://www.postgresql-sessions.org/_media/7/migration_sous_linux-transparents.pdf Exemple d'une migration de PostgreSQL sous Linux d'un système de suivi de production], Nicolas Relange<br />
* [http://www.postgresql-sessions.org/_media/7/pgsession_7_postgis_tips.pdf PostGIS : trucs et astuces ], Olivier Courtin<br />
* [http://www.postgresql-sessions.org/_media/7/tempus_light.pdf présentation du projet Tempus], Olivier Courtin<br />
* [http://www.postgresql-sessions.org/_media/7/office_.net_et_postgresql.pdf http://www.postgresql-sessions.org/_media/7/office_.net_et_postgresql.pdf Office, .NET & … PostgreSQL : tiercé gagnant pour les institutions publiques ?], Jean-Marc Guazzo<br />
* [http://www.postgresql-sessions.org/_media/7/postgresql95.pdf What's new in PostgreSQL 9.5 ?], Magnus Hagander<br />
* [http://www.postgresql-sessions.org/_media/7/presentation-pgsession-24092015_v8.4.pdf Témoignage utilisateur sur la mise en œuvre de la réplication], Michel Edwell<br />
* [http://www.postgresql-sessions.org/_media/7/powa_pgsession7.pdf Optimisation de requêtes, avec PoWA / pg_qualstats / pg_stat_kcache / HypoPG], Julien Rouhaud & Ronan Dunklau - Dalibo<br />
* [http://www.postgresql-sessions.org/_media/7/pgsessions7-oracle_comment_s_en_passer.pdf Oracle, comment peut-on s'en passer ?], Gilles Darold - Dalibo<br />
* [http://www.postgresql-sessions.org/_media/7/postgresql_conf.pdf Retour d'expérience sur le remplacement d'une base comme CouchBase par PostgreSQL, et son importance croissante dans les portfolios de bases de données de cette banque], Emmanuel Remy - CASDEN<br />
* [http://www.postgresql-sessions.org/_media/7/pg_day_pres_sodexi.pdf utilisateur de PostgreSQL depuis 2009], Fabrice Noël - SoDExI<br />
<br />
== Video ==<br />
<br />
Thanks to the Mozilla Foundation, [https://www.youtube.com/playlist?list=PLdz5EN2NV_7BXtGhlWNWepg0HCJ68KXRk the talks are available here].</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PostgreSQL_Related_Slides_and_Presentations&diff=25962PostgreSQL Related Slides and Presentations2015-09-30T14:48:13Z<p>Rjuju: </p>
<hr />
<div>Here you can find links to various PostgreSQL related presentations<br />
* [[pg_session_7|PostgreSQL Sessions #7 Sep, 2015]]<br />
* [[pgDay_Paris_2015|pgDay Paris 2015]]<br />
* [[PgConfUS_Talks_2015|PGConf US 2015]]<br />
* [[Nordic_PGDay_2015|Nordic PGDay, Copenhagen, 2015]]<br />
* [[FOSDEM_2015|FOSDEM, Brussels 2015]]<br />
* [http://vimeo.com/110472415 Level Up Conference 2014 | Craig Kersteins | Postgres: A Data Platform]<br />
* [[PostgreSQL_Conference_Europe_Talks_2014|PostgreSQL Conference Europe Talks 2014]]<br />
* [[PostgreSQL_Meetup_Paris_2014_Sept|PostgreSQL Meetup Paris, 2014-09-08]]<br />
* [[PGConf_NYC_2014_Talks|PGConf NYC 2014]]<br />
* [[Nordic_PGDay_2014|Nordic PGDay, Stockholm, 2014]]<br />
* [[PGDay_SCALE12x|PgDay SCALE 12x, 2014]]<br />
* [[FOSDEM_2014|FOSDEM, Brussels 2014]]<br />
* [[PGDay_Argentina_2013|PgDay Argentina 2013]]<br />
* [[German-Speaking PostgreSQL Conference 2013 | PGConf.DE 2013 - German-Speaking PostgreSQL Conference 2013]]<br />
* [[PostgreSQL_Conference_Europe_Talks_2013|PostgreSQL Conference Europe 2013]]<br />
* [[Postgres Open 2013]]<br />
* [[PGCon, Ottawa 2013]]<br />
* [[PGDay_NYC_2013_Talks|PGDay NYC 2013]]<br />
* [[FOSDEM_2013|FOSDEM, Brussels 2013]]<br />
* [[PGDay_Argentina_2012|PgDay Argentina 2012]]<br />
* [[PostgreSQL_Conference_Europe_Talks_2012|PostgreSQL Conference Europe 2012]]<br />
* [[Postgres Open 2012]]<br />
* [[German-Speaking PostgreSQL Conference 2011|PGConf.DE 2011]]<br />
* [[PGBR 2011, São Paulo]]<br />
* [[PostgreSQL_Conference_Europe_Talks_2011|PostgreSQL Conference Europe 2011]]<br />
* [[Postgres_Open_Talks_2011|Postgres Open 2011]]<br />
* [[FOSDEM, Brussels 2010]]<br />
* [[PGDay.EU, Paris 2009]]<br />
* [[PGCon, Ottawa 2009]]<br />
* [[FOSDEM, Brussels 2009]]<br />
* [http://www.arpug.com.ar/trac/wiki/PgDay2008 PgDay 2008 Rio de la Plata (Argentina)]<br />
* [[European PGDay 2008]]<br />
* [[Netways Nagios Conference 2008]]<br />
* [[FrOSCon, St. Augustin (near Bonn, Germany) 2008]]<br />
* [[PostgreSQLConferenceWest2008]]<br />
* [[PG UK 2008]]<br />
* [[FOSDEM, Brussels 2008]]<br />
* [http://postgresql.org.br/Palestras_do_PGCon_Brasil_2008 PGCon Brazil 2008]<br />
* [http://postgresql.org.br/Palestras_do_PGCon_Brasil_2007 PGCon Brazil 2007]<br />
* [[OSCon 2005]]<br />
* [[OSCon 2004]]<br />
* [[:Image:NUUG2005.pdf|NUUG 2005]]<br />
* [http://momjian.us/main/writings/computer.html Bruce Momjian Presentations]<br />
* [[PUG_Presentations]]<br />
* [http://www.postgresql.org/developer/coding PostgreSQL Coding]<br />
<br />
[[Category:Community]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=IRC2RWNames&diff=24588IRC2RWNames2015-04-20T09:11:57Z<p>Rjuju: </p>
<hr />
<div>=== List of IRC nicks with their respective real world names ===<br />
<br />
You can find many PostgreSQL users and developers chatting in [irc://irc.freenode.net/postgresql #postgresql on freenode]. Here's more information about some of the regulars there. '''Note:''' people are on the list below only when they want to be. Do not (re-)add anyone without their express permission.<br />
<br />
{| border="1"<br />
|-<br />
!Nickname || Real Name<br />
|-<br />
|ads || Andreas Scherbaum<br />
|-<br />
|agliodbs, aglio2 (freenode), jberkus (oftc) || Josh Berkus<br />
|-<br />
|ahammond || Andrew Hammond<br />
|-<br />
|alvherre || Alvaro Herrera<br />
|-<br />
|andres || Andres Freund<br />
|-<br />
|Assid || Satish Alwani<br />
|-<br />
|aurynn || Aurynn Shaw<br />
|-<br />
|BlueAidan/BlueAidan_work || [[user:davidblewett | David Blewett]]<br />
|-<br />
|bmomjian || Bruce Momjian<br />
|-<br />
|cbbrowne || Christopher Browne<br />
|-<br />
|cce || Clark C. Evans<br />
|-<br />
|chicagoben || Benjamin Johnson<br />
|-<br />
|crab || Abhijit Menon-Sen<br />
|-<br />
|Crad || Gavin M. Roy<br />
|- <br />
|daamien || Damien Clochard<br />
|-<br />
|DarcyB || Darcy Buskermolen<br />
|-<br />
|darkixion || Thom Brown<br />
|-<br />
|davidfetter || David Fetter<br />
|-<br />
|dbb || Brian M Hamlin / darkblue_b<br />
|-<br />
|dcolish || [http://www.unencrypted.org Dan Colish]<br />
|-<br />
|dcramer || Dave Cramer<br />
|-<br />
|DeciBull, TheCougar || Jim C. Nasby<br />
|-<br />
|dennisb || Dennis Bj&ouml;rklund<br />
|-<br />
|depesz || Hubert Lubaczewski<br />
|-<br />
|devrimgunduz || Devrim G&uuml;nd&uuml;z<br />
|-<br />
|digicon || [http://digicondev.blogspot.com Zach Conrad]<br />
|-<br />
|digitalknight || Atri Sharma<br />
|-<br />
|dim || [http://tapoueh.org Dimitri Fontaine]<br />
|-<br />
|direvus || Brendan Jurd<br />
|-<br />
|drbair || Ryan Bair<br />
|-<br />
|DrLou || Lou Picciano<br />
|-<br />
|duck_tape || Adi Alurkar<br />
|-<br />
|dvl || [http://langille.org/ Dan Langille]<br />
|-<br />
|eggyknap || Joshua Tolley<br />
|-<br />
|endpoint_david || David Christensen<br />
|-<br />
|eulerto || Euler Taveira<br />
|-<br />
|f3ew/devdas || Devdas Vasu Bhagat<br />
|-<br />
|feivel || Michael Meskes<br />
|-<br />
|elein || Elein Mustain<br />
|-<br />
|Gibheer || Stefan Radomski<br />
|-<br />
|gleu || Guillaume Lelarge<br />
|-<br />
|gorthx || [[User:Gabrielle|Gabrielle Roth]]<br />
|-<br />
|grzm || Michael Glaesemann<br />
|-<br />
|gsmet || Guillaume Smet<br />
|-<br />
|gregs1104 || Greg Smith<br />
|-<br />
|gurjeet || [[User:singh.gurjeet|Gurjeet Singh]]<br />
|-<br />
|G_SabinoMullane || Greg Sabino Mullane<br />
|-<br />
|HarrisonF || Harrison Fisk<br />
|-<br />
|ioguix || Jehan-Guillaume de Rorthais<br />
|-<br />
|indigo || Phil Frost<br />
|-<br />
|intgr || Marti Raudsepp<br />
|-<br />
|JanniCash || Jan Wieck<br />
|-<br />
|jconway || Joe Conway<br />
|-<br />
|jdavis, jdavis_ || Jeff Davis<br />
|-<br />
|jkatz05 || Jonathan S. Katz<br />
|-<br />
|johto || Marko Tiikkaja<br />
|-<br />
|jurka || Kris Jurka<br />
|-<br />
|justatheory || David Wheeler<br />
|-<br />
|jpa || Jean-Paul Argudo<br />
|-<br />
|jwp || James Pye<br />
|-<br />
|j_williams || Josh Williams<br />
|-<br />
|keithf4 || [http://www.keithf4.com Keith Fiske]<br />
|-<br />
|kgrittn || Kevin Grittner<br />
|-<br />
|klando || [[User:c2main|Cédric Villemain]]<br />
|-<br />
|larryrtx || Larry Rosenman<br />
|-<br />
|linuxpoet, postgresman || Joshua D. Drake<br />
|-<br />
|lluad || Steve Atkins<br />
|-<br />
|lsmith || Lukas Smith<br />
|-<br />
|macdice || Thomas Munro<br />
|-<br />
|mage_ || Julien Cigar<br />
|-<br />
|magnush || Magnus Hagander<br />
|-<br />
|maiku41 || Mike Blackwell<br />
|-<br />
|marco44 || Marc Cousin<br />
|-<br />
|markwkm || Mark Wong<br />
|-<br />
|mastermind || [[user:mastermind | Stefan Kaltenbrunner]]<br />
|-<br />
|mbalmer || [[user:mbalmer | Marc Balmer]]<br />
|-<br />
|merlin83 || Chua Khee Chin<br />
|-<br />
|merlinm || Merlin Moncure<br />
|-<br />
|metatrontech || Chris Travers<br />
|-<br />
|miracee || Susanne Ebrecht<br />
|-<br />
|Moosbert || Peter Eisentraut<br />
|-<br />
|Myon || [[user:Myon | Christoph Berg]]<br />
|-<br />
|neilc || Neil Conway<br />
|-<br />
|oicu || Andrew Dunstan<br />
|-<br />
|okbobcz || Pavel Stehule<br />
|-<br />
|patryk || Patryk Kordylewski<br />
|-<br />
|pasha_golub || [http://pgolub.wordpress.com/ Pavel Golub]<br />
|-<br />
|pg_docbot || [[IRCBotSyntax]]<br />
|-<br />
|pgSnake || Dave Page<br />
|-<br />
|PJMODOS || Petr Jel&iacute;nek<br />
|-<br />
|Possible || Robert Ivens<br />
|-<br />
|postwait || Theo Schlossnagle<br />
|-<br />
|prothid || R Brenton Strickler<br />
|-<br />
|psoo || Bernd Helmle<br />
|-<br />
|PSUdaemon || Phil Sorber<br />
|-<br />
|pyarra || Philip Yarra<br />
|-<br />
|raptelan || [[user:Cshobe|Casey Allen Shobe]]<br />
|-<br />
|rhaas || Robert Haas<br />
|-<br />
|RhodiumToad (formerly AndrewSN) || Andrew Gierth<br />
|-<br />
|rjuju || Julien Rouhaud<br />
|-<br />
|Robe || [[user:Robe | Michael Renner]]<br />
|-<br />
|rotellaro || Federico Campoli<br />
|-<br />
|rz || Kirill Simonov<br />
|-<br />
|SAS || Stéphane Schildknecht<br />
|-<br />
|schmiddy || Josh Kupershmidt<br />
|-<br />
|scrappy || Marc G. Fournier<br />
|-<br />
|sehrope || Sehrope Sarkuni<br />
|-<br />
|selenamarie || Selena Deckelmann<br />
|-<br />
|SkippyDigits || Sherri Kalm<br />
|-<br />
|Snow-Man || Stephen Frost<br />
|-<br />
|Spritz || Matteo Beccati<br />
|-<br />
|sternocera || Peter Geoghegan<br />
|-<br />
|StuckMojo, MojoWork || Jon Erdman<br />
|-<br />
|swm || Gavin Sherry<br />
|-<br />
|vy || Volkan YAZICI<br />
|-<br />
|wulczer || Jan Urbański<br />
|-<br />
|xaprb || Baron Schwartz<br />
|-<br />
|xocolatl || Vik Fearing<br />
|-<br />
|xzilla, xzi11a || [[User:Xzilla|Robert Treat]]<br />
|}<br />
<br />
[[Category:Community]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Monitoring&diff=24190Monitoring2015-02-02T21:25:00Z<p>Rjuju: </p>
<hr />
<div>== PostgreSQL builtin & contrib ==<br />
<br />
=== Statistics collector ===<br />
<br />
PostgreSQL collects lots of data on its own and offers it via the <tt>pg_stat(io)_</tt> system views<br />
<br />
* Official documentation on the [http://www.postgresql.org/docs/current/static/monitoring-stats.html Statistics Collector]<br />
* [http://www.varlena.com/GeneralBits/107.php Interpreting pg_stat Views]<br />
<br />
=== contrib extensions ===<br />
<br />
The following extensions offer access to Postgres internals which may be of interest or collect additional information. Most of them are shipped with Postgres (the <tt>-contrib</tt> packages may need to be installed) and can be activated via the [http://www.postgresql.org/docs/current/static/sql-createextension.html extension interface].<br />
<br />
==== pg_stat_statements ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstatstatements.html pg_stat_statements]</tt> tracks all queries that are executed on the server and records average runtime per query "class" among other parameters.<br />
<br />
==== pg_stat_plans ====<br />
<br />
<tt>[https://github.com/2ndQuadrant/pg_stat_plans pg_stat_plans]</tt> extends on pg_stat_statements and records query plans for all executed quries. This is very helpful when you're experiencing performance regressions due to inefficient query plans due to changed parameters or table sizes.<br />
<br />
==== pgstattuple ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstattuple.html pgstattuple]</tt> can generate statistics for tables and indexes, showing how much space in each table & index is consumed by live tuples, deleted tuples as well as how much unused space is available in each relation.<br />
<br />
==== pg_buffercache ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgbuffercache.html pg_buffercache]</tt> gives you introspection into Postgres' [http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-SHARED-BUFFERS shared buffers], showing how many pages of which relations are currently held in the cache.<br />
<br />
== External projects ==<br />
<br />
=== CLI tools ===<br />
<br />
==== pg_view ====<br />
<br />
<tt>[http://tech.zalando.com/getting-a-quick-view-of-your-postgresql-stats/ pg_view]</tt> is a Python-based tool to quickly get information about running databases and resources used by them as well as correlate running queries and why they might be slow.<br />
<br />
==== pg_activity ====<br />
<br />
<tt>[https://github.com/julmon/pg_activity pg_activity]</tt> is a htop like application for PostgreSQL server activity monitoring, written in Python.<br />
<br />
==== pgstats ====<br />
<br />
<tt>[https://github.com/gleu/pgstats pgstats]</tt> is a command line tool written in C which can sample various PostgreSQL informations. It also provides a tool to generate CSV files to graph the pgstats metrics.<br />
=== Checkers === <br />
<br />
==== check_postgres ====<br />
<br />
<tt>[http://bucardo.org/wiki/Check_postgres check_postgres]</tt> is a command line tool which is designed to be run from software like Icinga, MRTG or as a standalone tool. It can monitor many aspects of the database and trigger warnings when thresholds are violated.<br />
<br />
=== Interfaces & collectors ===<br />
<br />
These tools either offer an interface to PostgreSQL monitoring-relevant data or can aggregate and prepare them for other systems.<br />
<br />
==== pgsnmpd ====<br />
<br />
<tt>[http://pgsnmpd.projects.postgresql.org/ pgsnmpd]</tt> can run as a standalone SNMP server and implements the [http://www.faqs.org/rfcs/rfc1697.html RFC 1697 MIB] which is generic RDBMS [http://en.wikipedia.org/wiki/Management_information_base MIB]<br />
<br />
This is useful for network management systems which are limited to SNMP protocol.<br />
<br />
==== pganalyze-collector ====<br />
<br />
<tt>[https://github.com/pganalyze/pganalyze-collector pganalyze-collector]</tt> is a tool which collects <tt>pg_stat_plans</tt> query information as well as various performance-relevant database parameters and converts them into a JSON structure for easy ingestion in other systems.<br />
<br />
=== Generic monitoring solutions with plugins ===<br />
<br />
==== Munin ====<br />
<br />
PostgreSQL Plugins developed in Perl are included in the Core [http://munin-monitoring.org/ Munin] Distribution. The following plugins are included by default: <tt>postgres_bgwriter, postgres_locks_, postgres_tuples_, postgres_cache_, postgres_querylength_, postgres_users, postgres_checkpoints, postgres_scans_, postgres_xlog, postgres_connections_, postgres_size_, postgres_connections_db, postgres_transactions_</tt><br />
<br />
[http://aouyar.github.com/PyMunin/ PyMunin] includes a Multigraph Munin Plugin written in Python that implements the following graphs: <tt>pg_connections, pg_diskspace, pg_blockreads, pg_xact, pg_tup_read, pg_tup_write, pg_blockreads_detail, pg_xact_commit_detail, pg_xact_rollback_detail, pg_tup_return_detail, pg_tup_fetch_detail, pg_tup_delete_detail, pg_tup_update_detail, pg_tup_insert_detail</tt><br />
<br />
Detailed setup instructions for common Linux platforms can be found at [http://highperfpostgres.com/guides/postgresql-monitoring-with-munin/ highperfpostgres.com]<br />
<br />
==== Zabbix ====<br />
<br />
[http://pg-monz.github.io/pg_monz/index-en.html pg_monz] is a [http://www.zabbix.com/ Zabbix] monitoring template for Postgres.<br />
<br />
==== NewRelic ====<br />
<br />
[http://newrelic.com/ NewRelic] is a commercial SaaS application monitoring solution which offers a [https://newrelic.com/plugins/enterprisedb-corporation/30 PostgreSQL plugin] maintained by EnterpriseDB.<br />
<br />
==== Cacti ====<br />
<br />
There has been work done on building a Postgres template for [http://www.cacti.net/ Cacti], Details can be found at the [[Cacti]] page.<br />
<br />
=== Postgres-centric monitoring solutions ===<br />
<br />
==== EnterpriseDB Postgres Enterprise Manager ====<br />
<br />
[http://www.enterprisedb.com/products-services-training/products/postgres-enterprise-manager Postgres Enterprise Manager] is a commercial application offered by EnterpriseDB which covers many aspects of Postgres operations & monitoring in large environments.<br />
<br />
==== pganalyze ====<br />
<br />
[https://pganalyze.com/ pganalyze] is a commercial SaaS offering which focuses on performance monitoring and automated tuning suggestions.<br />
<br />
==== pgwatch ====<br />
<br />
[http://www.cybertec.at/en/products/pgwatch-cybertec-enterprise-postgresql-monitor/ pgwatch] is a PHP web application which offers interactive graphs for relevant Postgres data.<br />
<br />
==== pg_statsinfo & pg_stats_reporter ====<br />
<br />
[http://pgstatsinfo.projects.pgfoundry.org/ pg_statsinfo] is a Postgres extension that collects lots of performance-relevant information inside the Postgres server which then can be aggregated by pg_stats_reporter instances which provide a web interface to the collected data. Both are FOSS software maintained by NTT.<br />
<br />
==== PGObserver ====<br />
<br />
[http://zalando.github.io/PGObserver/ PGObserver] is a Python & Java-based Postgres monitoring solution developed by Zalando. It was developed with a focus on stored procedure performance but extended well beyond that. <br />
<br />
==== pgCluu ====<br />
<br />
[http://pgcluu.darold.net/ pgCluu] is a Perl-based monitoring solution which uses psql and [http://en.wikipedia.org/wiki/Sar_(Unix) sar] to collect information about Postgres servers and render comprehensive performance stats.<br />
<br />
==== PoWA ==== <br />
<br />
[http://dalibo.github.io/powa/ PoWA] is a PostgreSQL Workload Analyzer that gathers performance stats and provides real-time charts and graphs to help monitor and tune your PostgreSQL servers.<br />
<br />
==== OPM: Open PostgreSQL Monitoring ==== <br />
<br />
[http://opm.io Open PostgreSQL Monitoring (OPM)] is a free software suite designed to help you manage your PostgreSQL servers. It's a flexible tool that will follow the activity of each instance. It can gather stats, display dashboards and send warnings when something goes wrong. The long-term goal of the project is to provide similar features to those of Oracle Grid Control or SQL Server Management Studio.<br />
<br />
<br />
[[Category:Monitoring|!]]<br />
[[Category:Performance]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=Monitoring&diff=24119Monitoring2015-01-15T15:43:27Z<p>Rjuju: </p>
<hr />
<div>== PostgreSQL builtin & contrib ==<br />
<br />
=== Statistics collector ===<br />
<br />
PostgreSQL collects lots of data on its own and offers it via the <tt>pg_stat(io)_</tt> system views<br />
<br />
* Official documentation on the [http://www.postgresql.org/docs/current/static/monitoring-stats.html Statistics Collector]<br />
* [http://www.varlena.com/GeneralBits/107.php Interpreting pg_stat Views]<br />
<br />
=== contrib extensions ===<br />
<br />
The following extensions offer access to Postgres internals which may be of interest or collect additional information. Most of them are shipped with Postgres (the <tt>-contrib</tt> packages may need to be installed) and can be activated via the [http://www.postgresql.org/docs/current/static/sql-createextension.html extension interface].<br />
<br />
==== pg_stat_statements ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstatstatements.html pg_stat_statements]</tt> tracks all queries that are executed on the server and records average runtime per query "class" among other parameters.<br />
<br />
==== pg_stat_plans ====<br />
<br />
<tt>[https://github.com/2ndQuadrant/pg_stat_plans pg_stat_plans]</tt> extends on pg_stat_statements and records query plans for all executed quries. This is very helpful when you're experiencing performance regressions due to inefficient query plans due to changed parameters or table sizes.<br />
<br />
==== pgstattuple ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgstattuple.html pgstattuple]</tt> can generate statistics for tables and indexes, showing how much space in each table & index is consumed by live tuples, deleted tuples as well as how much unused space is available in each relation.<br />
<br />
==== pg_buffercache ====<br />
<br />
<tt>[http://www.postgresql.org/docs/current/static/pgbuffercache.html pg_buffercache]</tt> gives you introspection into Postgres' [http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-SHARED-BUFFERS shared buffers], showing how many pages of which relations are currently held in the cache.<br />
<br />
== External projects ==<br />
<br />
=== CLI tools ===<br />
<br />
==== pg_view ====<br />
<br />
<tt>[http://tech.zalando.com/getting-a-quick-view-of-your-postgresql-stats/ pg_view]</tt> is a Python-based tool to quickly get information about running databases and resources used by them as well as correlate running queries and why they might be slow.<br />
<br />
==== pgstats ====<br />
<br />
<tt>[https://github.com/gleu/pgstats pgstats]</tt> is a command line tool written in C which can sample various PostgreSQL informations. It also provides a tool to generate CSV files to graph the pgstats metrics.<br />
=== Checkers === <br />
<br />
==== check_postgres ====<br />
<br />
<tt>[http://bucardo.org/wiki/Check_postgres check_postgres]</tt> is a command line tool which is designed to be run from software like Icinga, MRTG or as a standalone tool. It can monitor many aspects of the database and trigger warnings when thresholds are violated.<br />
<br />
=== Interfaces & collectors ===<br />
<br />
These tools either offer an interface to PostgreSQL monitoring-relevant data or can aggregate and prepare them for other systems.<br />
<br />
==== pgsnmpd ====<br />
<br />
<tt>[http://pgsnmpd.projects.postgresql.org/ pgsnmpd]</tt> can run as a standalone SNMP server and implements the [http://www.faqs.org/rfcs/rfc1697.html RFC 1697 MIB] which is generic RDBMS [http://en.wikipedia.org/wiki/Management_information_base MIB]<br />
<br />
This is useful for network management systems which are limited to SNMP protocol.<br />
<br />
==== pganalyze-collector ====<br />
<br />
<tt>[https://github.com/pganalyze/pganalyze-collector pganalyze-collector]</tt> is a tool which collects <tt>pg_stat_plans</tt> query information as well as various performance-relevant database parameters and converts them into a JSON structure for easy ingestion in other systems.<br />
<br />
=== Generic monitoring solutions with plugins ===<br />
<br />
==== Munin ====<br />
<br />
PostgreSQL Plugins developed in Perl are included in the Core [http://munin-monitoring.org/ Munin] Distribution. The following plugins are included by default: <tt>postgres_bgwriter, postgres_locks_, postgres_tuples_, postgres_cache_, postgres_querylength_, postgres_users, postgres_checkpoints, postgres_scans_, postgres_xlog, postgres_connections_, postgres_size_, postgres_connections_db, postgres_transactions_</tt><br />
<br />
[http://aouyar.github.com/PyMunin/ PyMunin] includes a Multigraph Munin Plugin written in Python that implements the following graphs: <tt>pg_connections, pg_diskspace, pg_blockreads, pg_xact, pg_tup_read, pg_tup_write, pg_blockreads_detail, pg_xact_commit_detail, pg_xact_rollback_detail, pg_tup_return_detail, pg_tup_fetch_detail, pg_tup_delete_detail, pg_tup_update_detail, pg_tup_insert_detail</tt><br />
<br />
Detailed setup instructions for common Linux platforms can be found at [http://highperfpostgres.com/guides/postgresql-monitoring-with-munin/ highperfpostgres.com]<br />
<br />
==== Zabbix ====<br />
<br />
[http://pg-monz.github.io/pg_monz/index-en.html pg_monz] is a [http://www.zabbix.com/ Zabbix] monitoring template for Postgres.<br />
<br />
==== NewRelic ====<br />
<br />
[http://newrelic.com/ NewRelic] is a commercial SaaS application monitoring solution which offers a [https://newrelic.com/plugins/enterprisedb-corporation/30 PostgreSQL plugin] maintained by EnterpriseDB.<br />
<br />
==== Cacti ====<br />
<br />
There has been work done on building a Postgres template for [http://www.cacti.net/ Cacti], Details can be found at the [[Cacti]] page.<br />
<br />
=== Postgres-centric monitoring solutions ===<br />
<br />
==== EnterpriseDB Postgres Enterprise Manager ====<br />
<br />
[http://www.enterprisedb.com/products-services-training/products/postgres-enterprise-manager Postgres Enterprise Manager] is a commercial application offered by EnterpriseDB which covers many aspects of Postgres operations & monitoring in large environments.<br />
<br />
==== pganalyze ====<br />
<br />
[https://pganalyze.com/ pganalyze] is a commercial SaaS offering which focuses on performance monitoring and automated tuning suggestions.<br />
<br />
==== pgwatch ====<br />
<br />
[http://www.cybertec.at/en/products/pgwatch-cybertec-enterprise-postgresql-monitor/ pgwatch] is a PHP web application which offers interactive graphs for relevant Postgres data.<br />
<br />
==== pg_statsinfo & pg_stats_reporter ====<br />
<br />
[http://pgstatsinfo.projects.pgfoundry.org/ pg_statsinfo] is a Postgres extension that collects lots of performance-relevant information inside the Postgres server which then can be aggregated by pg_stats_reporter instances which provide a web interface to the collected data. Both are FOSS software maintained by NTT.<br />
<br />
==== PGObserver ====<br />
<br />
[http://zalando.github.io/PGObserver/ PGObserver] is a Python & Java-based Postgres monitoring solution developed by Zalando. It was developed with a focus on stored procedure performance but extended well beyond that. <br />
<br />
==== pgCluu ====<br />
<br />
[http://pgcluu.darold.net/ pgCluu] is a Perl-based monitoring solution which uses psql and [http://en.wikipedia.org/wiki/Sar_(Unix) sar] to collect information about Postgres servers and render comprehensive performance stats.<br />
<br />
==== PoWA ==== <br />
<br />
[http://dalibo.github.io/powa/ PoWA] is a PostgreSQL Workload Analyzer that gathers performance stats and provides real-time charts and graphs to help monitor and tune your PostgreSQL servers.<br />
<br />
==== OPM: Open PostgreSQL Monitoring ==== <br />
<br />
[http://opm.io Open PostgreSQL Monitoring (OPM)] is a free software suite designed to help you manage your PostgreSQL servers. It's a flexible tool that will follow the activity of each instance. It can gather stats, display dashboards and send warnings when something goes wrong. The long-term goal of the project is to provide similar features to those of Oracle Grid Control or SQL Server Management Studio.<br />
<br />
<br />
[[Category:Monitoring|!]]<br />
[[Category:Performance]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=PGDay_FOSDEM_2013&diff=18811PGDay FOSDEM 20132013-01-07T09:51:48Z<p>Rjuju: /* Dinner */</p>
<hr />
<div>= PGDay FOSDEM 2013 =<br />
<br />
This is the first PgDay we hold in Belgium. FOSDEM PGDay 2013 will be held on Feb 1st in Brussels, Belgium, at the Radisson Blu Royal hotel. As an extension to the regular PostgreSQL devroom at FOSDEM, it will cover topics for PostgreSQL users, developers and contributors, and anybody else interested in PostgreSQL<br />
<br />
== Details ==<br />
<br />
* '''Date:''' Feb 01st, 2013 9am-5pm<br />
* '''Venue:''': Radisson Blu Royal Hotel<br />
* '''Coordinator:''': PostgreSQL Europe [mailto:contact@pgconf.eu contact@pgconf.eu]<br />
* '''Website:''': http://fosdem2013.pgconf.eu/<br />
<br />
== Registration ==<br />
<br />
Free attendance, web registration required: http://fosdem2013.pgconf.eu/registration/ (limited seats)<br />
<br />
== Schedule ==<br />
<br />
Schedule is be published at: http://fosdem2013.pgconf.eu/schedule/<br />
<br />
== Location and Venue ==<br />
<br />
http://fosdem2013.pgconf.eu/venue/<br />
<br />
Address: <br />
<br />
http://www.radissonblu.com/royalhotel-brussels/location<br />
<br />
<br />
== Dinner ==<br />
<br />
We are organizing a dinner after the event on Friday 1st, 2013 at Hard Rock Cafe Brussels. We have limited (30) number of seats, so please add your name to this list before going there.<br />
<br />
If you are bringing someone to the event, make sure you enter your name *twice* (or more) on the list, so the attendee count matches!<br />
<br />
Attendees:<br />
<br />
# Devrim Gündüz<br />
# Devrim Gündüz +1<br />
# Magnus Hagander<br />
# Andreas Scherbaum<br />
# Andreas Scherbaum +1<br />
# Jean-Paul Argudo<br />
# Patryk Kordylewski<br />
# Patryk Kordylewski +1<br />
# Dimitri Fontaine<br />
# Julien Rouhaud<br />
<br />
[[Category:PostgreSQL Events]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=What%27s_new_in_PostgreSQL_9.2&diff=18084What's new in PostgreSQL 9.22012-08-24T07:30:27Z<p>Rjuju: </p>
<hr />
<div>{{Languages}}<br />
<br />
This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release &ndash; PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.<br />
<br />
<br />
=Major new features=<br />
<br />
==Index-only scans <!-- Robert Haas, Ibrar Ahmed, Heikki Linnakangas, Tom Lane -->==<br />
<br />
In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.<br />
<br />
It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.<br />
<br />
There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.<br />
<br />
The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).<br />
<br />
Here is an example.<br />
<br />
CREATE TABLE demo_ios (col1 float, col2 float, col3 text);<br />
<br />
In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains wont be that big in real life.<br />
<br />
INSERT INTO demo_ios SELECT generate_series(1,100000000),random(), 'mynotsolongstring';<br />
<br />
SELECT pg_size_pretty(pg_total_relation_size('demo_ios'));<br />
pg_size_pretty <br />
----------------<br />
6512 MB<br />
<br />
Let's pretend that the query is this:<br />
<br />
SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.01 AND 0.02<br />
<br />
In order to use an index only scan on this query, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).<br />
<br />
CREATE index idx_demo_ios on demo_ios(col2,col1);<br />
<br />
We vacuum the visibility map to be up-to-date:<br />
<br />
VACUUM demo_ios;<br />
<br />
All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).<br />
<br />
Let's first try without Index Only Scans:<br />
<br />
SET enable_indexonlyscan to off;<br />
<br />
EXPLAIN (analyze,buffers) select col1,col2 FROM demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------------------<br />
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)<br />
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Rows Removed by Index Recheck: 68098621<br />
Buffers: shared hit=2 read=587779<br />
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Buffers: shared hit=2 read=3835<br />
Total runtime: 364390.127 ms<br />
<br />
<br />
With Index Only Scans:<br />
<br />
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------------------------------<br />
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Heap Fetches: 0<br />
Buffers: shared hit=923073 read=3848<br />
Total runtime: 4297.405 ms<br />
<br />
<br />
<br />
As nothing is free, there are a few things to keep in mind:<br />
<br />
* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.<br />
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.<br />
* Gains will probably be smaller in real life situations.<br />
<br />
This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.<br />
<br />
==Replication improvements <!-- Fujii Masao, Simon Riggs, Magnus Hagander, Jun Ishizuka -->==<br />
<br />
Streaming Replication is getting even more polished with this release. One on the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources.<br />
<br />
Moreover, in case of a failover, it was very complicated to reconnect all the remaining slaves to the newly promoted master.<br />
<br />
To be on the safe side, it was easier to re-synchronize the slaves to the new masters from scratch, meaning that during this failover, only one server was active, and under heavy load, as it was used to rebuild all the slaves.<br />
<br />
* With 9.2, a slave can also be a replication master, allowing cascading replication.<br />
<br />
Let's build this. We start with an already working 9.2 database.<br />
<br />
We set it up for replication:<br />
<br />
postgresql.conf:<br />
wal_level=hot_standby #(could be archive too)<br />
max_wal_senders=5<br />
hot_standby=on<br />
<br />
You'll probably also want to activate archiving in production, it won't be done here.<br />
<br />
pg_hba.conf (do not use trust in production):<br />
host replication replication_user 0.0.0.0/0 md5<br />
<br />
Create the user:<br />
create user replication_user replication password 'secret';<br />
<br />
Clone the database:<br />
<br />
pg_basebackup -h localhost -U replication_user -D data2<br />
Password:<br />
<br />
We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf):<br />
port=5433<br />
<br />
We add a recovery.conf to tell it how to stream from the master database:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret' <br />
<br />
pg_ctl -D data2 start<br />
server starting<br />
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9D0000B8<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, let's add a second slave, which will use this slave:<br />
<br />
<br />
pg_basebackup -h localhost -U replication_user -D data3 -p 5433<br />
Password: <br />
<br />
We edit data3's postgresql.conf to change the port:<br />
port=5434<br />
<br />
We modify the recovery.conf to stream from the slave:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'<br />
<br />
We start the cluster:<br />
pg_ctl -D data3 start<br />
server starting<br />
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST<br />
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9E000000<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).<br />
<br />
<br />
* As you may have noticed from the examble, pg_basebackup now works from slaves.<br />
<br />
* There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?<br />
<br />
pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:<br />
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user<br />
<br />
will connect to the master (or a slave), and start creating files: <br />
ls /tmp/new_logs/<br />
00000001000000000000009E.partial<br />
<br />
Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.<br />
Remember to rename the last segment to remove the .partial suffix before using it with PITR or other.<br />
<br />
* synchronous_commit has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, some people will be interested in this compromise.<br />
<br />
<br />
<br />
<br />
==JSON datatype==<br />
<br />
The JSON datatype is meant for storing JSON-structured data. It will validate that the input JSON string is correct JSON:<br />
<br />
=# SELECT '{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}'::json;<br />
json <br />
-------------------------------------------------------------------<br />
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}<br />
(1 row)<br />
<br />
=# SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;<br />
ERROR: invalid input syntax for type json at character 8<br />
DETAIL: Expected ":", but found ",".<br />
CONTEXT: JSON data, line 1: {"username",...<br />
STATEMENT: SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;<br />
ERROR: invalid input syntax for type json<br />
LINE 1: SELECT '{"username","posts":121,"emailaddress":"john@nowhere...<br />
^<br />
DETAIL: Expected ":", but found ",".<br />
CONTEXT: JSON data, line 1: {"username",...<br />
<br />
You can also convert a row type to JSON:<br />
<br />
=#SELECT * FROM demo ;<br />
username | posts | emailaddress <br />
----------+-------+---------------------<br />
john | 121 | john@nowhere.com<br />
mickael | 215 | mickael@nowhere.com<br />
(2 rows)<br />
<br />
=# SELECT row_to_json(demo) FROM demo;<br />
row_to_json <br />
-------------------------------------------------------------------------<br />
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}<br />
{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}<br />
(2 rows)<br />
<br />
Or an array type:<br />
<br />
<br />
=# select array_to_json(array_agg(demo)) from demo;<br />
array_to_json <br />
---------------------------------------------------------------------------------------------------------------------------------------------<br />
[{"username":"john","posts":121,"emailaddress":"john@nowhere.com"},{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}]<br />
(1 row)<br />
<br />
== Range Types ==<br />
<br />
Range types are used to store a range of data of a given type. There are a few pre-defined types. They are integer (int4range), bigint (int8range), numeric (numrange), timestamp without timezone (tsrange), timestamp with timezone (tstzrange), and date (daterange).<br />
<br />
Ranges can be made of continuous (numeric, timestamp...) or discrete (integer, date...) data types. They can be open (the bound isn't part of the range) or closed (the bound is part of the range). A bound can also be infinite.<br />
<br />
Without these datatypes, most people solve the range problems by using two columns in a table. These range types are much more powerful, as you can use many operators on them:<br />
<br />
Here is the intersection between then 1000(open)-2000(closed) and 1000(closed)-1200(closed) numeric range:<br />
<br />
SELECT '(1000,2000]'::numrange * '[1000,1200]'::numrange;<br />
?column? <br />
-------------<br />
(1000,1200]<br />
(1 row)<br />
<br />
So you can query on things like: «give me all ranges that intersect this»:<br />
<br />
=# SELECT * from test_range ;<br />
period <br />
-----------------------------------------------------<br />
["2012-01-01 00:00:00+01","2012-01-02 12:00:00+01"]<br />
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]<br />
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]<br />
(3 rows)<br />
<br />
<br />
=# SELECT * FROM test_range WHERE period && '[2012-01-03 00:00:00,2012-01-03 12:00:00]'; <br />
period <br />
-----------------------------------------------------<br />
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]<br />
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]<br />
(2 rows)<br />
<br />
This query could use an index defined like this:<br />
<br />
=# CREATE INDEX idx_test_range on test_range USING gist (period);<br />
<br />
<br />
<br />
You can also use these range data types to define exclusion constraints:<br />
<br />
CREATE EXTENSION btree_gist ;<br />
CREATE TABLE reservation (room_id int, period tstzrange);<br />
ALTER TABLE reservation ADD EXCLUDE USING GIST (room_id WITH =, period WITH &&);<br />
<br />
This means that now it is forbidden to have two records in this table where room_id is equal and period overlaps. The extension btree_gist is required to create a GiST index on room_id (it's an integer, it is usually indexed with a btree index).<br />
<br />
<br />
=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');<br />
INSERT 0 1<br />
=# INSERT INTO reservation VALUES (2,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');<br />
INSERT 0 1<br />
=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');<br />
ERROR: conflicting key value violates exclusion constraint "reservation_room_id_period_excl"<br />
DETAIL: Key (room_id, period)=(1, ("2012-08-23 14:45:00+02","2012-08-23 15:15:00+02")) <br />
conflicts with existing key (room_id, period)=(1, ("2012-08-23 14:00:00+02","2012-08-23 15:00:00+02")).<br />
STATEMENT: INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');<br />
<br />
One can also declare new range types.<br />
<br />
=Performance improvements=<br />
<br />
This version has performance improvements on a very large range of domains (non-exaustive):<br />
<br />
* The most visible will probably be the Index Only Scans, which has already been introduced in this document.<br />
<br />
* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. <!-- Robert Haas --><br />
<br />
* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. <!-- Peter Geoghegan --><br />
<br />
* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption <!--Peter Geoghegan-->. This is especially useful on virtualized and embedded environments.<br />
<br />
* COPY has been improved, it will generate less WAL volume and less locks of tables's pages. <!-- Heikki Linnakangas --><br />
<br />
* Statistics are collected on array contents <!-- Alexander Korotkov -->, allowing for better estimations of selectivity on array operations.<br />
<br />
* The system can now track IO durations <!--Ants Aasma --><br />
<br />
This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.<br />
<br />
If you are interested in this timing data, it's better to first check if your system will support it without to much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:<br />
<br />
<pre><br />
$ pg_test_timing <br />
Testing timing overhead for 3 seconds.<br />
Per loop time including overhead: 28.02 nsec<br />
Histogram of timing durations:<br />
< usec: count percent<br />
32: 41 0.00004%<br />
16: 1405 0.00131%<br />
8: 200 0.00019%<br />
4: 388 0.00036%<br />
2: 2982558 2.78523%<br />
1: 104100166 97.21287%<br />
</pre><br />
<br />
Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].<br />
<br />
Anyway, here is the data you'll be able to collect if your system is ready for this:<br />
<br />
First, you'll get per-database statistics, which will now give accurate informations about which database is doing most I/O:<br />
<br />
<pre><br />
=# SELECT * FROM pg_stat_database WHERE datname = 'mydb';<br />
-[ RECORD 1 ]--+------------------------------<br />
datid | 16384<br />
datname | mydb<br />
numbackends | 1<br />
xact_commit | 270<br />
xact_rollback | 2<br />
blks_read | 1961<br />
blks_hit | 17944<br />
tup_returned | 269035<br />
tup_fetched | 8850<br />
tup_inserted | 16<br />
tup_updated | 4<br />
tup_deleted | 45<br />
conflicts | 0<br />
temp_files | 0<br />
temp_bytes | 0<br />
deadlocks | 0<br />
blk_read_time | 583.774<br />
blk_write_time | 0<br />
stats_reset | 2012-07-03 17:18:54.796817+02<br />
</pre><br />
We see here that mydb has only consumed 583.774 milliseconds of read time.<br />
<br />
Explain will benefit from this too:<br />
<pre><br />
=# EXPLAIN (analyze,buffers) SELECT count(*) FROM mots ;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------<br />
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
Total runtime: 22.059 ms<br />
</pre><br />
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).<br />
<br />
And last, if you have enabled pg_stat_statements:<br />
<pre><br />
select * from pg_stat_statements where query ~ 'words';<br />
-[ RECORD 1 ]-------+---------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | select count(*) from words;<br />
calls | 2<br />
total_time | 78.332<br />
rows | 2<br />
shared_blks_hit | 0<br />
shared_blks_read | 986<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 58.427<br />
blk_write_time | 0<br />
</pre><br />
<br />
* As for every version, the optimizer has received its share of improvements <!-- Tom Lane--><br />
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.<br />
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it would have been efficient.<br />
<br />
This example is straight from the developpers mailing lists <!-- Andres Freund -->:<br />
<br />
<pre><br />
CREATE TABLE a (<br />
a_id serial PRIMARY KEY NOT NULL,<br />
b_id integer<br />
);<br />
CREATE INDEX a__b_id ON a USING btree (b_id);<br />
<br />
<br />
CREATE TABLE b (<br />
b_id serial NOT NULL,<br />
c_id integer<br />
);<br />
CREATE INDEX b__c_id ON b USING btree (c_id);<br />
<br />
<br />
CREATE TABLE c (<br />
c_id serial PRIMARY KEY NOT NULL,<br />
value integer UNIQUE<br />
);<br />
<br />
INSERT INTO b (b_id, c_id)<br />
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO a(b_id)<br />
SELECT g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO c(c_id,value)<br />
VALUES (1,1);<br />
</pre><br />
<br />
So we have a referencing b, b referencing c.<br />
<br />
Here is an example of a query working badly with PostgreSQL 9.1:<br />
<br />
<pre><br />
EXPLAIN ANALYZE SELECT 1 <br />
FROM <br />
c<br />
WHERE<br />
EXISTS (<br />
SELECT * <br />
FROM a<br />
JOIN b USING (b_id)<br />
WHERE b.c_id = c.c_id)<br />
AND c.value = 1;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)<br />
Join Filter: (c.c_id = b.c_id)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)<br />
Hash Cond: (a.b_id = b.b_id)<br />
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)<br />
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)<br />
Buckets: 8192 Batches: 1 Memory Usage: 1954kB<br />
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)<br />
Total runtime: 13.842 ms<br />
</pre><br />
<br />
Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.<br />
<br />
Here's what 9.2 does with this query:<br />
<br />
<pre><br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
The «parameterized path» is:<br />
<pre><br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.<br />
<br />
This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.<br />
<br />
=SP-GiST=<br />
<br />
SP-GiST stands for Space Partitionned GiST, GiST being Generalized Search Tree. GiST is an index type, and has been available for quite a while in PostgreSQL. GiST is already very efficient at indexing complex data types, but performance tends to suffer when the source data isn't uniformly distributed. SP-GiST tries to fix that.<br />
<br />
As all indexing methods available in PostgreSQL, SP-GiST is a generic indexing method, meaning its purpose is to index whatever you'll throw at it, using operators you'll provide. It means that if you want to create a new datatype, and make it indexable through SP-GiST, you'll have to follow the documented API.<br />
<br />
SP-GiST can be used to implement 3 type of indexes: trie (suffix) indexing, Quadtree (data is divided into quadrants), and k-d tree (k-dimensional tree).<br />
<br />
For now, SP-GiST is provided with operator families called "quad_point_ops", "kd_point_ops" and "text_ops".<br />
<br />
As their names indicate, the first one indexes point types, using a quadtree, the second one indexes point types using a k-d tree, and the third one indexes text, using suffix.<br />
<br />
=pg_stat_statements=<br />
<br />
This contrib module has received a lot of improvements in this version:<br />
<br />
* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.<br />
<br />
<pre><br />
=#SELECT * FROM words WHERE word= 'foo';<br />
word <br />
------<br />
(0 ligne)<br />
<br />
=# SELECT * FROM words WHERE word= 'bar';<br />
word <br />
------<br />
bar<br />
<br />
=#select * from pg_stat_statements where query like '%words where%';<br />
-[ RECORD 1 ]-------+-----------------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | SELECT * FROM words WHERE word= ?;<br />
calls | 2<br />
total_time | 142.314<br />
rows | 1<br />
shared_blks_hit | 3<br />
shared_blks_read | 5<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 142.165<br />
blk_write_time | 0<br />
<br />
</pre><br />
<br />
The two queries are shown as one in pg_stat_statements.<br />
<br />
* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That way it is easier to use, and avoids the double-counting there was with PostgreSQL 9.1.<br />
<br />
* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.<br />
<br />
= Explain improvements=<br />
<br />
* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive <!--Tomas Vondra--><br />
<br />
=# EXPLAIN (analyze on,timing off) SELECT * FROM reservation ;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------<br />
Seq Scan on reservation (cost=0.00..22.30 rows=1230 width=36) (actual rows=2 loops=1)<br />
Total runtime: 0.045 ms<br />
<br />
<br />
<br />
* Have EXPLAIN ANALYZE report the number of rows rejected by filter steps <!--(Marko Tiikkaja)--><br />
<br />
This new feature makes it much easier to know how many rows are removed by a filter (and spot potential places to put indexes):<br />
<br />
=# EXPLAIN ANALYZE SELECT * FROM test WHERE a ~ 'tra';<br />
QUERY PLAN <br />
---------------------------------------------------------------------------------------------------------------<br />
Seq Scan on test (cost=0.00..106876.56 rows=2002 width=11) (actual time=2.914..8538.285 rows=120256 loops=1)<br />
Filter: (a ~ 'tra'::text)<br />
Rows Removed by Filter: 5905600<br />
Total runtime: 8549.539 ms<br />
(4 rows)<br />
<br />
=Backward compatibility=<br />
<br />
These changes may incur regressions in your applications.<br />
<br />
==Ensure that xpath() escapes special characters in string values <!-- (Florian Pflug)--> ==<br />
<br />
Before 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
<<br />
<br />
'<' Isn't valid XML.<br />
</pre><br />
With 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
&amp;lt;<br />
</pre><br />
<br />
==Remove hstore's => operator <!-- (Robert Haas)-->==<br />
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.<br />
<br />
In 9.1:<br />
<pre><br />
=# SELECT 'a'=>'b';<br />
?column? <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# SELECT pg_typeof('a'=>'b');<br />
pg_typeof <br />
-----------<br />
hstore<br />
(1 row)<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown at character 11<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
STATEMENT: SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown<br />
LINE 1: SELECT 'a'=>'b';<br />
^<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
</pre><br />
<br />
It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:<br />
<br />
<pre><br />
=# select hstore('a=>b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# select hstore('a','b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
</pre><br />
are still two valid ways to input a hstore.<br />
<br />
"=>" is removed as an operator as it is a reserved keyword in SQL.<br />
<br />
<br />
==Have pg_relation_size() and friends return NULL if the object does not exist <!-- (Phil Sorber)-->==<br />
<br />
A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.<br />
<br />
<br />
==Remove the spclocation field from pg_tablespace <!-- (Magnus Hagander)-->==<br />
<br />
The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.<br />
<br />
So this field has been removed. To get the tablespace's location, use pg_tablespace_location():<br />
<br />
<pre><br />
=# SELECT *, pg_tablespace_location(oid) AS spclocation FROM pg_tablespace;<br />
spcname | spcowner | spcacl | spcoptions | spclocation <br />
------------+----------+--------+------------+----------------<br />
pg_default | 10 | | | <br />
pg_global | 10 | | | <br />
tmptblspc | 10 | | | /tmp/tmptblspc<br />
</pre><br />
<br />
==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight <!-- (Tom Lane) -->==<br />
<br />
<br />
With PostgreSQL 9.1:<br />
<br />
<pre><br />
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
<br />
=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
<br />
There is no difference in behaviour between a timstamp with or without timezone.<br />
<br />
With 9.1:<br />
<pre><br />
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341187200<br />
(1 row)<br />
<br />
=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.<br />
<br />
<br />
==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 <!-- (Bruce Momjian)-->==<br />
<br />
The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.<br />
<br />
Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.<br />
<br />
With 9.1:<br />
<pre><br />
=# SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
1200-07-02<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
2200-07-02<br />
</pre><br />
<br />
<br />
==pg_stat_activity and pg_stat_replication's definitions have changed <!--Magnus Hagander -->==<br />
<br />
The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:<br />
<br />
* current_query disappears and is replaced by two columns:<br />
** state: is the session running a query, waiting<br />
** query: what is the last run (or still running if stat is "active") query<br />
* The column procpid is renamed to pid, to be consistent with other system views<br />
<br />
The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.<br />
<br />
Here is an example:<br />
<pre><br />
-[ RECORD 1 ]----+---------------------------------<br />
datid | 16384<br />
datname | postgres<br />
pid | 20804<br />
usesysid | 10<br />
usename | postgres<br />
application_name | psql<br />
client_addr | <br />
client_hostname | <br />
client_port | -1<br />
backend_start | 2012-07-02 15:02:51.146427+02<br />
xact_start | 2012-07-02 15:15:28.386865+02<br />
query_start | 2012-07-02 15:15:30.410834+02<br />
state_change | 2012-07-02 15:15:30.411287+02<br />
waiting | f<br />
state | idle in transaction<br />
query | DELETE FROM test;<br />
</pre><br />
<br />
With PostgreSQL 9.1, all we would have would be «idle in transaction».<br />
<br />
As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.<br />
The view pg_stat_replication has also changed. The column procpid is renamed to pid, to also be consistent with other system views.<br />
<br />
==Change all SQL-level statistics timing values to float8-stored milliseconds <!-- (Tom Lane) -->==<br />
<br />
pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.<br />
<br />
==postgresql.conf parameters changes <!-- (Heikki Linnakangas, Tom Lane, Peter Eisentraut) -->==<br />
<br />
* silent_mode has been removed. Use pg_ctl -l postmaster.log<br />
* wal_sender_delay has been removed. It is no longer needed<br />
* custom_variable_classes has been removed. All «classes» are accepted without declaration now<br />
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files<br />
<br />
= Other new features =<br />
<br />
== DROP INDEX CONCURRENTLY ==<br />
<br />
The regular DROP INDEX command takes an exclusive lock on the table. Most of the time, this isn't a problem, because this lock is short-lived. The problem usually occurs when:<br />
<br />
* A long-running transaction is running, and has a (shared) lock on the table<br />
* A DROP INDEX is run on this table in another session, asking for an exclusive lock (and waiting for it, as it won't be granted until the long-running transaction ends)<br />
<br />
At this point, all other transactions needing to take a shared lock on the table (for a simple SELECT for instance) will have to wait too: their lock acquisition is queued after the DROP INDEX's one.<br />
<br />
<br />
DROP INDEX CONCURRENTLY works around this and won't lock normal DML statements, just as CREATE INDEX CONCURRENTLY. The main limitation is the same: DROP INDEX CONCURRENTLY can't be run in a transaction. Moreover, you can only DROP one index with CONCURRENTLY, and CASCADE isn't supported either.<br />
<br />
== NOT VALID CHECK constraints ==<br />
<br />
PostgreSQL 9.1 introduced «NOT VALID» foreign keys. This has been extended to CHECK constraints. Adding a «NOT VALID» constraint on a table means that current data won't be validated, only new and updated rows.<br />
<br />
=# CREATE TABLE test (a int); <br />
CREATE TABLE<br />
=# INSERT INTO test SELECT generate_series(1,100);<br />
INSERT 0 100<br />
=# ALTER TABLE test ADD CHECK (a>100) NOT VALID;<br />
ALTER TABLE<br />
=# INSERT INTO test VALUES (99);<br />
ERROR: new row for relation "test" violates check constraint "test_a_check"<br />
DETAIL: Failing row contains (99).<br />
=# INSERT INTO test VALUES (101);<br />
INSERT 0 1<br />
<br />
Then, later, we can validate the whole table:<br />
<br />
=# ALTER TABLE test VALIDATE CONSTRAINT test_a_check ;<br />
ERROR: check constraint "test_a_check" is violated by some row<br />
<br />
Domains, which are types with added constraints, can also be declared as not valid, and validated later.<br />
<br />
Check constraints can also be renamed now:<br />
<br />
=# ALTER TABLE test RENAME CONSTRAINT test_a_check TO validate_a;<br />
ALTER TABLE<br />
<br />
Last, but not least, constraints can be declared as not inheritable, which will be useful in partitioned environments. Let's take PostgreSQL documentation example, and see how it improves the situation:<br />
<br />
CREATE TABLE measurement (<br />
city_id int not null,<br />
logdate date not null,<br />
peaktemp int,<br />
unitsales int,<br />
CHECK (logdate IS NULL) NO INHERIT<br />
);<br />
<br />
CREATE TABLE measurement_y2006m02 (<br />
CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )<br />
) INHERITS (measurement);<br />
CREATE TABLE measurement_y2006m03 (<br />
CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' )<br />
) INHERITS (measurement);<br />
<br />
<br />
INSERT INTO measurement VALUES (1,'2006-02-20',1,1);<br />
ERROR: new row for relation "measurement" violates check constraint "measurement_logdate_check"<br />
DETAIL: Failing row contains (1, 2006-02-20, 1, 1).<br />
INSERT INTO measurement_y2006m02 VALUES (1,'2006-02-20',1,1);<br />
INSERT 0 1<br />
<br />
Until now, every check constraint created on measurement would have been inherited by children tables. So adding a constraint forbidding inserts, or allowing only some of them, on the parent table was impossible.<br />
<br />
== Reduce ALTER TABLE rewrites ==<br />
<br />
A table won't get rewritten anymore during an ALTER TABLE when changing the type of a column in the following cases:<br />
<br />
* varchar(x) to varchar(y) when y > x. It works too if going from varchar(x) to varchar or text (no size limitation)<br />
* numeric(x,z) to numeric(y,z) when y>x, or to numeric without specifier<br />
* varbit(x) to varbit(y) when y>x, or to varbit without specifier<br />
* timestamp(x) to timestamp(y) when y>x or timestamp without specifier<br />
* timestamptz(x) to timestamptz(y) when y>x or timestamptz without specifier<br />
* interval(x) to interval(y) when y>x or interval without specifier<br />
<br />
== Security barriers and Leakproof ==<br />
<br />
This new feature has to do with views security. First, let's explain the problem, with a very simplified example:<br />
<br />
=# CREATE TABLE all_data (company_id int, company_data varchar);<br />
CREATE TABLE<br />
# INSERT INTO all_data VALUES (1,'secret_data_for_company_1');<br />
INSERT 0 1<br />
=# INSERT INTO all_data VALUES (2,'secret_data_for_company_2');<br />
INSERT 0 1<br />
=# CREATE VIEW company1_data AS SELECT * FROM all_data WHERE company_id = 1;<br />
CREATE VIEW<br />
<br />
This is a quite classical way of giving access to only a part of a table to a user: we'll create a user for company_id 1, grant to him the right to access company1_data, and deny him the right to access all_data.<br />
<br />
The plan to this query is the following:<br />
<br />
=# explain SELECT * FROM company1_data ;<br />
QUERY PLAN <br />
----------------------------------------------------------<br />
Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)<br />
Filter: (company_id = 1)<br />
<br />
Even if there was more data, a sequential scan could still be forced: just "SET enable_indexscan to OFF" and the likes.<br />
<br />
So this query reads all the records from all_data, filters them, and returns to the user only the matching rows. There is a way to display scanned records before they are filtered: just create a function with a very low cost, and call it while doing the query:<br />
<br />
CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LANGUAGE plpgsql AS<br />
$$<br />
BEGIN<br />
RAISE NOTICE '%',$1;<br />
RETURN true;<br />
END<br />
$$<br />
COST 0.1;<br />
<br />
This function just has to cost less than the = operator, which costs 1, to be executed first.<br />
<br />
The result is this:<br />
<br />
<br />
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);<br />
NOTICE: secret_data_for_company_1<br />
NOTICE: secret_data_for_company_2<br />
company_id | company_data <br />
------------+---------------------------<br />
1 | secret_data_for_company_1<br />
(1 row)<br />
<br />
We got access to the record from the second company (in the NOTICE messages).<br />
<br />
So this is the first new feature: the view can be declared as implementing "security barriers":<br />
<br />
<br />
=# CREATE VIEW company1_data WITH (security_barrier) AS SELECT * FROM all_data WHERE company_id = 1;<br />
CREATE VIEW<br />
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);<br />
NOTICE: secret_data_for_company_1<br />
company_id | company_data <br />
------------+---------------------------<br />
1 | secret_data_for_company_1<br />
(1 row)<br />
<br />
The view's not leaking anymore. The problem, of course, is that there is a performance impact: maybe the "peek" function could have made the query faster, by filtering lots of rows early in the plan.<br />
<br />
This leads to the complementary feature: some function may be declared as "LEAKPROOF", meaning that they won't leak the data they are passed into error or notice messages.<br />
<br />
Declaring our peek function as LEAKPROOF is a very bad idea, but let's do it just to demonstrate how it's used:<br />
<br />
CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LEAKPROOF LANGUAGE plpgsql AS<br />
$$<br />
BEGIN<br />
RAISE NOTICE '%',$1;<br />
RETURN true;<br />
END<br />
$$<br />
COST 0.1;<br />
<br />
A LEAKPROOF function is executed «normally»:<br />
<br />
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);<br />
NOTICE: secret_data_for_company_1<br />
NOTICE: secret_data_for_company_2<br />
company_id | company_data <br />
------------+---------------------------<br />
1 | secret_data_for_company_1<br />
(1 row)<br />
<br />
Of course, in our case, peek isn't LEAKPROOF and shouldn't be declared as such. Only superuser have the permission to declare a LEAKPROOF function.<br />
<br />
== New options for pg_dump ==<br />
<br />
Until now, one could ask pg_dump to dump a table's data, or a table's meta-data (DDL statements for creating the table's structure, indexes, constraints). Some meta-data is better restored before the data (the table's structure, check constraints), some is better after the data (indexes, unique constraints, foreign keys…), for performance reasons mostly.<br />
<br />
So there are now a few more options:<br />
<br />
* --section=pre-data: dump what's needed before restoring the data. Of course, this can be combined with a -t for instance, to specify one table<br />
* --section=post-data : dump what's needed after restoring the data.<br />
* --section=data: dump the data<br />
* --exclude-table-data: dump everything, except THIS table's data. It means pg_dump will still dump other tables' data.<br />
<br />
<br />
<br />
<br />
<br />
[[Category:PostgreSQL 9.2]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=What%27s_new_in_PostgreSQL_9.2&diff=18000What's new in PostgreSQL 9.22012-08-10T13:40:22Z<p>Rjuju: /* pg_stat_activity and pg_stat_replication's definitions has changed */</p>
<hr />
<div>{{Languages}}<br />
<br />
This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release &ndash; PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.<br />
<br />
'''This page is incomplete!'''<br />
<br />
=Major new features=<br />
<br />
==Index-only scans <!-- Robert Haas, Ibrar Ahmed, Heikki Linnakangas, Tom Lane -->==<br />
<br />
In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.<br />
<br />
It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.<br />
<br />
There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.<br />
<br />
The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).<br />
<br />
Here is an example.<br />
<br />
create table demo_ios (col1 float, col2 float, col3 text);<br />
<br />
In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains wont be that big in real life.<br />
<br />
insert into demo_ios select generate_series(1,100000000),random(), 'mynotsolongstring';<br />
<br />
select pg_size_pretty(pg_total_relation_size('demo_ios'));<br />
pg_size_pretty <br />
----------------<br />
6512 MB<br />
<br />
Let's pretend that the query is this:<br />
<br />
SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.02 AND 0.03<br />
<br />
In order to use an index only scan on this, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).<br />
<br />
CREATE index idx_demo_ios on demo_ios(col2,col1);<br />
<br />
We vacuum the visibility map to be up-to-date:<br />
<br />
VACUUM demo_ios;<br />
<br />
All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).<br />
<br />
Let's first try without Index Only Scans:<br />
<br />
set enable_indexonlyscan to off;<br />
<br />
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------------------<br />
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)<br />
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Rows Removed by Index Recheck: 68098621<br />
Buffers: shared hit=2 read=587779<br />
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Buffers: shared hit=2 read=3835<br />
Total runtime: 364390.127 ms<br />
<br />
<br />
With Index Only Scans:<br />
<br />
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------------------------------<br />
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Heap Fetches: 0<br />
Buffers: shared hit=923073 read=3848<br />
Total runtime: 4297.405 ms<br />
<br />
<br />
<br />
As nothing is free, there are a few things to note:<br />
<br />
* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.<br />
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.<br />
* Gains will probably be smaller in real life situations.<br />
<br />
This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.<br />
<br />
==Replication improvements <!-- Fujii Masao, Simon Riggs, Magnus Hagander, Jun Ishizuka -->==<br />
<br />
Streaming Replication is getting even more polished with this release. One on the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources.<br />
<br />
Moreover, in case of a failover, it was very complicated to reconnect all the remaining slaves to the newly promoted master.<br />
<br />
To be on the safe side, it was easier to re-synchronize the slaves to the new masters from scratch, meaning that during this failover, only one server was active, and under heavy load, as it was used to rebuild all the slaves.<br />
<br />
* With 9.2, a slave can also be a replication master, allowing for cascading replication.<br />
<br />
Let's build this. We start with an already working 9.2 database.<br />
<br />
We set it up for replication:<br />
<br />
postgresql.conf:<br />
wal_level=hot_standby #(could be archive too)<br />
max_wal_senders=5<br />
hot_standby=on<br />
<br />
You'll probably also want to activate archiving in production, it won't be done here.<br />
<br />
pg_hba.conf (do not use trust in production):<br />
host replication replication_user 0.0.0.0/0 md5<br />
<br />
Create the user:<br />
create user replication_user replication password 'secret';<br />
<br />
Clone the database:<br />
<br />
pg_basebackup -h localhost -U replication_user -D data2<br />
Password:<br />
<br />
We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf):<br />
port=5433<br />
<br />
We add a recovery.conf to tell it how to stream from the master database:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret' <br />
<br />
pg_ctl -D data2 start<br />
server starting<br />
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9D0000B8<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, let's add a second slave, which will use this slave:<br />
<br />
<br />
pg_basebackup -h localhost -U replication_user -D data3 -p 5433<br />
Password: <br />
<br />
We edit data3's postgresql.conf to change the port:<br />
port=5434<br />
<br />
We modify the recovery.conf to stream from the slave:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'<br />
<br />
We start the cluster:<br />
pg_ctl -D data3 start<br />
server starting<br />
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST<br />
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9E000000<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).<br />
<br />
<br />
* As you may have noticed from the examble, pg_basebackup now works from slaves.<br />
<br />
* There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?<br />
<br />
pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:<br />
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user<br />
<br />
will connect to the master (or a slave), and start creating files: <br />
ls /tmp/new_logs/<br />
00000001000000000000009E.partial<br />
<br />
Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.<br />
<br />
* synchronous_commit has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, some people will be interested in this compromise.<br />
<br />
<br />
<br />
<br />
==JSON datatype==<br />
The JSON datatype is meant for storing JSON-structured data. (More info: [http://www.depesz.com/2012/02/12/waiting-for-9-2-json/ depesz blog])<br />
<br />
== Range Types ==<br />
[[RangeTypes]] are added.<br />
(More info: [http://www.depesz.com/2011/11/07/waiting-for-9-2-range-data-types/])<br />
<br />
=Performance improvements=<br />
<br />
This version has performance improvements on a very large range of domains (non-exaustive):<br />
<br />
* The most visible will probably be the Index Only Scans, which has already been introduced in this document.<br />
<br />
* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. <!-- Robert Haas --><br />
<br />
* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. <!-- Peter Geoghegan --><br />
<br />
* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption <!--Peter Geoghegan-->. This is especially useful on virtualized and embedded environments.<br />
<br />
* COPY has been improved, it will generate less WAL volume and less locks of tables's pages. <!-- Heikki Linnakangas --><br />
<br />
* Statistics are collected on array contents <!-- Alexander Korotkov -->, allowing for better estimations of selectivity on array operations.<br />
<br />
* The system can now track IO durations <!--Ants Aasma --><br />
<br />
This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.<br />
<br />
If you are interested in this timing data, it's better to first check if your system will support it without to much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:<br />
<br />
<pre><br />
$ pg_test_timing <br />
Testing timing overhead for 3 seconds.<br />
Per loop time including overhead: 28.02 nsec<br />
Histogram of timing durations:<br />
< usec: count percent<br />
32: 41 0.00004%<br />
16: 1405 0.00131%<br />
8: 200 0.00019%<br />
4: 388 0.00036%<br />
2: 2982558 2.78523%<br />
1: 104100166 97.21287%<br />
</pre><br />
<br />
Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].<br />
<br />
Anyway, here is the data you'll be able to collect if your system is ready for this:<br />
<br />
First, you'll get per-database statistics, which will now give accurate information about which database is doing most I/O:<br />
<br />
<pre><br />
=# select * from pg_stat_database where datname = 'mydb';<br />
-[ RECORD 1 ]--+------------------------------<br />
datid | 16384<br />
datname | mydb<br />
numbackends | 1<br />
xact_commit | 270<br />
xact_rollback | 2<br />
blks_read | 1961<br />
blks_hit | 17944<br />
tup_returned | 269035<br />
tup_fetched | 8850<br />
tup_inserted | 16<br />
tup_updated | 4<br />
tup_deleted | 45<br />
conflicts | 0<br />
temp_files | 0<br />
temp_bytes | 0<br />
deadlocks | 0<br />
blk_read_time | 583.774<br />
blk_write_time | 0<br />
stats_reset | 2012-07-03 17:18:54.796817+02<br />
</pre><br />
We see here that mydb has only consumed 583.774 milliseconds of read time.<br />
<br />
Explain will benefit from this too:<br />
<pre><br />
=# explain (analyze,buffers) select count(*) from mots ;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------<br />
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
Total runtime: 22.059 ms<br />
</pre><br />
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).<br />
<br />
And last, if you have enabled pg_stat_statements:<br />
<pre><br />
select * from pg_stat_statements where query ~ 'words';<br />
-[ RECORD 1 ]-------+---------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | select count(*) from words;<br />
calls | 2<br />
total_time | 78.332<br />
rows | 2<br />
shared_blks_hit | 0<br />
shared_blks_read | 986<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 58.427<br />
blk_write_time | 0<br />
</pre><br />
<br />
* As for every version, the optimizer has received its share of improvements <!-- Tom Lane--><br />
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.<br />
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it wanted to.<br />
<br />
This example is straight from the developpers mailing lists <!-- Andres Freund -->:<br />
<br />
<pre><br />
CREATE TABLE a (<br />
a_id serial PRIMARY KEY NOT NULL,<br />
b_id integer<br />
);<br />
CREATE INDEX a__b_id ON a USING btree (b_id);<br />
<br />
<br />
CREATE TABLE b (<br />
b_id serial NOT NULL,<br />
c_id integer<br />
);<br />
CREATE INDEX b__c_id ON b USING btree (c_id);<br />
<br />
<br />
CREATE TABLE c (<br />
c_id serial PRIMARY KEY NOT NULL,<br />
value integer UNIQUE<br />
);<br />
<br />
INSERT INTO b (b_id, c_id)<br />
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO a(b_id)<br />
SELECT g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO c(c_id,value)<br />
VALUES (1,1);<br />
</pre><br />
<br />
So we have a referencing b, b referencing c.<br />
<br />
Here is an example of a query working badly with PostgreSQL 9.1:<br />
<br />
<pre><br />
EXPLAIN ANALYZE SELECT 1 <br />
FROM <br />
c<br />
WHERE<br />
EXISTS (<br />
SELECT * <br />
FROM a<br />
JOIN b USING (b_id)<br />
WHERE b.c_id = c.c_id)<br />
AND c.value = 1;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)<br />
Join Filter: (c.c_id = b.c_id)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)<br />
Hash Cond: (a.b_id = b.b_id)<br />
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)<br />
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)<br />
Buckets: 8192 Batches: 1 Memory Usage: 1954kB<br />
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)<br />
Total runtime: 13.842 ms<br />
</pre><br />
<br />
Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.<br />
<br />
Here's what 9.2 does with this query:<br />
<br />
<pre><br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
The «parameterized path» is:<br />
<pre><br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.<br />
<br />
This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.<br />
<br />
<br />
=SP-GIST=<br />
TODO<br />
<br />
=pg_stat_statements=<br />
<br />
This contrib module has received a lot of improvements in this version:<br />
<br />
* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.<br />
<br />
<pre><br />
=#select * from words where word= 'foo';<br />
word <br />
------<br />
(0 ligne)<br />
<br />
=# select * from words where word= 'bar';<br />
word <br />
------<br />
bar<br />
<br />
=#select * from pg_stat_statements where query like '%words where%';<br />
-[ RECORD 1 ]-------+-----------------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | select * from words where word= ?;<br />
calls | 2<br />
total_time | 142.314<br />
rows | 1<br />
shared_blks_hit | 3<br />
shared_blks_read | 5<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 142.165<br />
blk_write_time | 0<br />
<br />
</pre><br />
<br />
The two queries are shown as one in pg_stat_statements.<br />
<br />
* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That way it is easier to use, and avoids the double-counting there was with PostgreSQL 9.1.<br />
<br />
* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.<br />
<br />
= Explain improvements=<br />
<br />
* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive <!--Tomas Vondra--><br />
<br />
<br />
Have EXPLAIN ANALYZE report the number of rows rejected by filter steps (Marko Tiikkaja)<br />
<br />
=Backward compatibility=<br />
<br />
These changes may incur regressions in your applications.<br />
<br />
==Ensure that xpath() escapes special characters in string values <!-- (Florian Pflug)--> ==<br />
<br />
Before 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
<<br />
<br />
'<' Isn't valid XML.<br />
</pre><br />
With 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
&amp;lt;<br />
</pre><br />
<br />
==Remove hstore's => operator <!-- (Robert Haas)-->==<br />
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.<br />
<br />
In 9.1:<br />
<pre><br />
SELECT pg_typeof('a'=>'b');<br />
pg_typeof <br />
-----------<br />
hstore<br />
(1 row)<br />
<br />
=# SELECT 'a'=>'b';<br />
?column? <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# SELECT pg_typeof('a'=>'b');<br />
pg_typeof <br />
-----------<br />
hstore<br />
(1 row)<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown at character 11<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
STATEMENT: SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown<br />
LINE 1: SELECT 'a'=>'b';<br />
^<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
</pre><br />
<br />
It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:<br />
<br />
<pre><br />
=# select hstore('a=>b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# select hstore('a','b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
</pre><br />
are still two valid ways to input a hstore.<br />
<br />
"=>" is removed as an operator as it is a reserved keyword in SQL.<br />
<br />
<br />
==Have pg_relation_size() and friends return NULL if the object does not exist <!-- (Phil Sorber)-->==<br />
<br />
A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.<br />
<br />
<br />
==Remove the spclocation field from pg_tablespace <!-- (Magnus Hagander)-->==<br />
<br />
The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.<br />
<br />
So this field has been removed. To get the tablespace's location, use pg_tablespace_location():<br />
<br />
<pre><br />
=# select *, pg_tablespace_location(oid) as spclocation from pg_tablespace;<br />
spcname | spcowner | spcacl | spcoptions | spclocation <br />
------------+----------+--------+------------+----------------<br />
pg_default | 10 | | | <br />
pg_global | 10 | | | <br />
tmptblspc | 10 | | | /tmp/tmptblspc<br />
</pre><br />
<br />
==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight <!-- (Tom Lane) -->==<br />
<br />
<br />
With PostgreSQL 9.1:<br />
<br />
<pre><br />
=#SELECT extract(epoch from '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
<br />
=# SELECT extract(epoch from '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
<br />
There is no difference in behaviour between a timstamp with or without timezone.<br />
<br />
With 9.1:<br />
<pre><br />
=#SELECT extract(epoch from '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341187200<br />
(1 row)<br />
<br />
=# SELECT extract(epoch from '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.<br />
<br />
<br />
==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 <!-- (Bruce Momjian)-->==<br />
<br />
The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.<br />
<br />
Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.<br />
<br />
With 9.1:<br />
<pre><br />
=# SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
1200-07-02<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
2200-07-02<br />
</pre><br />
<br />
<br />
==pg_stat_activity and pg_stat_replication's definitions have changed <!--Magnus Hagander -->==<br />
<br />
The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:<br />
<br />
* current_query disappears and is replaced by two columns:<br />
** state: is the session running a query, waiting<br />
** query: what is the last run (or still running) query<br />
* The column procpid is renamed to pid, to be consistent with other system views<br />
<br />
The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.<br />
<br />
Here is an example:<br />
<pre><br />
-[ RECORD 1 ]----+---------------------------------<br />
datid | 16384<br />
datname | postgres<br />
pid | 20804<br />
usesysid | 10<br />
usename | postgres<br />
application_name | psql<br />
client_addr | <br />
client_hostname | <br />
client_port | -1<br />
backend_start | 2012-07-02 15:02:51.146427+02<br />
xact_start | 2012-07-02 15:15:28.386865+02<br />
query_start | 2012-07-02 15:15:30.410834+02<br />
state_change | 2012-07-02 15:15:30.411287+02<br />
waiting | f<br />
state | idle in transaction<br />
query | DELETE FROM test;<br />
</pre><br />
<br />
With PostgreSQL 9.1, all we would have would be «idle in transaction».<br />
<br />
As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.<br />
The view pg_stat_replication has also changes. The column procpid is rename to pid, to also be consistent with other system views.<br />
<br />
==Change all SQL-level statistics timing values to float8-stored milliseconds <!-- (Tom Lane) -->==<br />
<br />
pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.<br />
<br />
==postgresql.conf parameters changes <!-- (Heikki Linnakangas, Tom Lane, Peter Eisentraut) -->==<br />
<br />
* silent_mode has been removed. Use pg_ctl -l postmaster.log<br />
* wal_sender_delay has been removed. It is no longer needed<br />
* custom_variable_classes has been removed. All «classes» are accepted without declaration now<br />
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files<br />
<br />
[[Category:PostgreSQL 9.2]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=What%27s_new_in_PostgreSQL_9.2&diff=17999What's new in PostgreSQL 9.22012-08-10T13:39:54Z<p>Rjuju: /* Backward compatibility */</p>
<hr />
<div>{{Languages}}<br />
<br />
This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release &ndash; PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.<br />
<br />
'''This page is incomplete!'''<br />
<br />
=Major new features=<br />
<br />
==Index-only scans <!-- Robert Haas, Ibrar Ahmed, Heikki Linnakangas, Tom Lane -->==<br />
<br />
In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.<br />
<br />
It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.<br />
<br />
There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.<br />
<br />
The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).<br />
<br />
Here is an example.<br />
<br />
create table demo_ios (col1 float, col2 float, col3 text);<br />
<br />
In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains wont be that big in real life.<br />
<br />
insert into demo_ios select generate_series(1,100000000),random(), 'mynotsolongstring';<br />
<br />
select pg_size_pretty(pg_total_relation_size('demo_ios'));<br />
pg_size_pretty <br />
----------------<br />
6512 MB<br />
<br />
Let's pretend that the query is this:<br />
<br />
SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.02 AND 0.03<br />
<br />
In order to use an index only scan on this, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).<br />
<br />
CREATE index idx_demo_ios on demo_ios(col2,col1);<br />
<br />
We vacuum the visibility map to be up-to-date:<br />
<br />
VACUUM demo_ios;<br />
<br />
All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).<br />
<br />
Let's first try without Index Only Scans:<br />
<br />
set enable_indexonlyscan to off;<br />
<br />
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------------------<br />
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)<br />
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Rows Removed by Index Recheck: 68098621<br />
Buffers: shared hit=2 read=587779<br />
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Buffers: shared hit=2 read=3835<br />
Total runtime: 364390.127 ms<br />
<br />
<br />
With Index Only Scans:<br />
<br />
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------------------------------<br />
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Heap Fetches: 0<br />
Buffers: shared hit=923073 read=3848<br />
Total runtime: 4297.405 ms<br />
<br />
<br />
<br />
As nothing is free, there are a few things to note:<br />
<br />
* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.<br />
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.<br />
* Gains will probably be smaller in real life situations.<br />
<br />
This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.<br />
<br />
==Replication improvements <!-- Fujii Masao, Simon Riggs, Magnus Hagander, Jun Ishizuka -->==<br />
<br />
Streaming Replication is getting even more polished with this release. One on the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources.<br />
<br />
Moreover, in case of a failover, it was very complicated to reconnect all the remaining slaves to the newly promoted master.<br />
<br />
To be on the safe side, it was easier to re-synchronize the slaves to the new masters from scratch, meaning that during this failover, only one server was active, and under heavy load, as it was used to rebuild all the slaves.<br />
<br />
* With 9.2, a slave can also be a replication master, allowing for cascading replication.<br />
<br />
Let's build this. We start with an already working 9.2 database.<br />
<br />
We set it up for replication:<br />
<br />
postgresql.conf:<br />
wal_level=hot_standby #(could be archive too)<br />
max_wal_senders=5<br />
hot_standby=on<br />
<br />
You'll probably also want to activate archiving in production, it won't be done here.<br />
<br />
pg_hba.conf (do not use trust in production):<br />
host replication replication_user 0.0.0.0/0 md5<br />
<br />
Create the user:<br />
create user replication_user replication password 'secret';<br />
<br />
Clone the database:<br />
<br />
pg_basebackup -h localhost -U replication_user -D data2<br />
Password:<br />
<br />
We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf):<br />
port=5433<br />
<br />
We add a recovery.conf to tell it how to stream from the master database:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret' <br />
<br />
pg_ctl -D data2 start<br />
server starting<br />
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9D0000B8<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, let's add a second slave, which will use this slave:<br />
<br />
<br />
pg_basebackup -h localhost -U replication_user -D data3 -p 5433<br />
Password: <br />
<br />
We edit data3's postgresql.conf to change the port:<br />
port=5434<br />
<br />
We modify the recovery.conf to stream from the slave:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'<br />
<br />
We start the cluster:<br />
pg_ctl -D data3 start<br />
server starting<br />
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST<br />
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9E000000<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).<br />
<br />
<br />
* As you may have noticed from the examble, pg_basebackup now works from slaves.<br />
<br />
* There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?<br />
<br />
pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:<br />
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user<br />
<br />
will connect to the master (or a slave), and start creating files: <br />
ls /tmp/new_logs/<br />
00000001000000000000009E.partial<br />
<br />
Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.<br />
<br />
* synchronous_commit has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, some people will be interested in this compromise.<br />
<br />
<br />
<br />
<br />
==JSON datatype==<br />
The JSON datatype is meant for storing JSON-structured data. (More info: [http://www.depesz.com/2012/02/12/waiting-for-9-2-json/ depesz blog])<br />
<br />
== Range Types ==<br />
[[RangeTypes]] are added.<br />
(More info: [http://www.depesz.com/2011/11/07/waiting-for-9-2-range-data-types/])<br />
<br />
=Performance improvements=<br />
<br />
This version has performance improvements on a very large range of domains (non-exaustive):<br />
<br />
* The most visible will probably be the Index Only Scans, which has already been introduced in this document.<br />
<br />
* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. <!-- Robert Haas --><br />
<br />
* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. <!-- Peter Geoghegan --><br />
<br />
* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption <!--Peter Geoghegan-->. This is especially useful on virtualized and embedded environments.<br />
<br />
* COPY has been improved, it will generate less WAL volume and less locks of tables's pages. <!-- Heikki Linnakangas --><br />
<br />
* Statistics are collected on array contents <!-- Alexander Korotkov -->, allowing for better estimations of selectivity on array operations.<br />
<br />
* The system can now track IO durations <!--Ants Aasma --><br />
<br />
This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.<br />
<br />
If you are interested in this timing data, it's better to first check if your system will support it without to much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:<br />
<br />
<pre><br />
$ pg_test_timing <br />
Testing timing overhead for 3 seconds.<br />
Per loop time including overhead: 28.02 nsec<br />
Histogram of timing durations:<br />
< usec: count percent<br />
32: 41 0.00004%<br />
16: 1405 0.00131%<br />
8: 200 0.00019%<br />
4: 388 0.00036%<br />
2: 2982558 2.78523%<br />
1: 104100166 97.21287%<br />
</pre><br />
<br />
Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].<br />
<br />
Anyway, here is the data you'll be able to collect if your system is ready for this:<br />
<br />
First, you'll get per-database statistics, which will now give accurate information about which database is doing most I/O:<br />
<br />
<pre><br />
=# select * from pg_stat_database where datname = 'mydb';<br />
-[ RECORD 1 ]--+------------------------------<br />
datid | 16384<br />
datname | mydb<br />
numbackends | 1<br />
xact_commit | 270<br />
xact_rollback | 2<br />
blks_read | 1961<br />
blks_hit | 17944<br />
tup_returned | 269035<br />
tup_fetched | 8850<br />
tup_inserted | 16<br />
tup_updated | 4<br />
tup_deleted | 45<br />
conflicts | 0<br />
temp_files | 0<br />
temp_bytes | 0<br />
deadlocks | 0<br />
blk_read_time | 583.774<br />
blk_write_time | 0<br />
stats_reset | 2012-07-03 17:18:54.796817+02<br />
</pre><br />
We see here that mydb has only consumed 583.774 milliseconds of read time.<br />
<br />
Explain will benefit from this too:<br />
<pre><br />
=# explain (analyze,buffers) select count(*) from mots ;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------<br />
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
Total runtime: 22.059 ms<br />
</pre><br />
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).<br />
<br />
And last, if you have enabled pg_stat_statements:<br />
<pre><br />
select * from pg_stat_statements where query ~ 'words';<br />
-[ RECORD 1 ]-------+---------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | select count(*) from words;<br />
calls | 2<br />
total_time | 78.332<br />
rows | 2<br />
shared_blks_hit | 0<br />
shared_blks_read | 986<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 58.427<br />
blk_write_time | 0<br />
</pre><br />
<br />
* As for every version, the optimizer has received its share of improvements <!-- Tom Lane--><br />
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.<br />
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it wanted to.<br />
<br />
This example is straight from the developpers mailing lists <!-- Andres Freund -->:<br />
<br />
<pre><br />
CREATE TABLE a (<br />
a_id serial PRIMARY KEY NOT NULL,<br />
b_id integer<br />
);<br />
CREATE INDEX a__b_id ON a USING btree (b_id);<br />
<br />
<br />
CREATE TABLE b (<br />
b_id serial NOT NULL,<br />
c_id integer<br />
);<br />
CREATE INDEX b__c_id ON b USING btree (c_id);<br />
<br />
<br />
CREATE TABLE c (<br />
c_id serial PRIMARY KEY NOT NULL,<br />
value integer UNIQUE<br />
);<br />
<br />
INSERT INTO b (b_id, c_id)<br />
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO a(b_id)<br />
SELECT g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO c(c_id,value)<br />
VALUES (1,1);<br />
</pre><br />
<br />
So we have a referencing b, b referencing c.<br />
<br />
Here is an example of a query working badly with PostgreSQL 9.1:<br />
<br />
<pre><br />
EXPLAIN ANALYZE SELECT 1 <br />
FROM <br />
c<br />
WHERE<br />
EXISTS (<br />
SELECT * <br />
FROM a<br />
JOIN b USING (b_id)<br />
WHERE b.c_id = c.c_id)<br />
AND c.value = 1;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)<br />
Join Filter: (c.c_id = b.c_id)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)<br />
Hash Cond: (a.b_id = b.b_id)<br />
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)<br />
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)<br />
Buckets: 8192 Batches: 1 Memory Usage: 1954kB<br />
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)<br />
Total runtime: 13.842 ms<br />
</pre><br />
<br />
Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.<br />
<br />
Here's what 9.2 does with this query:<br />
<br />
<pre><br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
The «parameterized path» is:<br />
<pre><br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.<br />
<br />
This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.<br />
<br />
<br />
=SP-GIST=<br />
TODO<br />
<br />
=pg_stat_statements=<br />
<br />
This contrib module has received a lot of improvements in this version:<br />
<br />
* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.<br />
<br />
<pre><br />
=#select * from words where word= 'foo';<br />
word <br />
------<br />
(0 ligne)<br />
<br />
=# select * from words where word= 'bar';<br />
word <br />
------<br />
bar<br />
<br />
=#select * from pg_stat_statements where query like '%words where%';<br />
-[ RECORD 1 ]-------+-----------------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | select * from words where word= ?;<br />
calls | 2<br />
total_time | 142.314<br />
rows | 1<br />
shared_blks_hit | 3<br />
shared_blks_read | 5<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 142.165<br />
blk_write_time | 0<br />
<br />
</pre><br />
<br />
The two queries are shown as one in pg_stat_statements.<br />
<br />
* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That way it is easier to use, and avoids the double-counting there was with PostgreSQL 9.1.<br />
<br />
* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.<br />
<br />
= Explain improvements=<br />
<br />
* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive <!--Tomas Vondra--><br />
<br />
<br />
Have EXPLAIN ANALYZE report the number of rows rejected by filter steps (Marko Tiikkaja)<br />
<br />
=Backward compatibility=<br />
<br />
These changes may incur regressions in your applications.<br />
<br />
==Ensure that xpath() escapes special characters in string values <!-- (Florian Pflug)--> ==<br />
<br />
Before 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
<<br />
<br />
'<' Isn't valid XML.<br />
</pre><br />
With 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
&amp;lt;<br />
</pre><br />
<br />
==Remove hstore's => operator <!-- (Robert Haas)-->==<br />
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.<br />
<br />
In 9.1:<br />
<pre><br />
SELECT pg_typeof('a'=>'b');<br />
pg_typeof <br />
-----------<br />
hstore<br />
(1 row)<br />
<br />
=# SELECT 'a'=>'b';<br />
?column? <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# SELECT pg_typeof('a'=>'b');<br />
pg_typeof <br />
-----------<br />
hstore<br />
(1 row)<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown at character 11<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
STATEMENT: SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown<br />
LINE 1: SELECT 'a'=>'b';<br />
^<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
</pre><br />
<br />
It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:<br />
<br />
<pre><br />
=# select hstore('a=>b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# select hstore('a','b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
</pre><br />
are still two valid ways to input a hstore.<br />
<br />
"=>" is removed as an operator as it is a reserved keyword in SQL.<br />
<br />
<br />
==Have pg_relation_size() and friends return NULL if the object does not exist <!-- (Phil Sorber)-->==<br />
<br />
A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.<br />
<br />
<br />
==Remove the spclocation field from pg_tablespace <!-- (Magnus Hagander)-->==<br />
<br />
The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.<br />
<br />
So this field has been removed. To get the tablespace's location, use pg_tablespace_location():<br />
<br />
<pre><br />
=# select *, pg_tablespace_location(oid) as spclocation from pg_tablespace;<br />
spcname | spcowner | spcacl | spcoptions | spclocation <br />
------------+----------+--------+------------+----------------<br />
pg_default | 10 | | | <br />
pg_global | 10 | | | <br />
tmptblspc | 10 | | | /tmp/tmptblspc<br />
</pre><br />
<br />
==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight <!-- (Tom Lane) -->==<br />
<br />
<br />
With PostgreSQL 9.1:<br />
<br />
<pre><br />
=#SELECT extract(epoch from '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
<br />
=# SELECT extract(epoch from '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
<br />
There is no difference in behaviour between a timstamp with or without timezone.<br />
<br />
With 9.1:<br />
<pre><br />
=#SELECT extract(epoch from '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341187200<br />
(1 row)<br />
<br />
=# SELECT extract(epoch from '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.<br />
<br />
<br />
==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 <!-- (Bruce Momjian)-->==<br />
<br />
The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.<br />
<br />
Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.<br />
<br />
With 9.1:<br />
<pre><br />
=# SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
1200-07-02<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
2200-07-02<br />
</pre><br />
<br />
<br />
==pg_stat_activity and pg_stat_replication's definitions has changed <!--Magnus Hagander -->==<br />
<br />
The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:<br />
<br />
* current_query disappears and is replaced by two columns:<br />
** state: is the session running a query, waiting<br />
** query: what is the last run (or still running) query<br />
* The column procpid is renamed to pid, to be consistent with other system views<br />
<br />
The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.<br />
<br />
Here is an example:<br />
<pre><br />
-[ RECORD 1 ]----+---------------------------------<br />
datid | 16384<br />
datname | postgres<br />
pid | 20804<br />
usesysid | 10<br />
usename | postgres<br />
application_name | psql<br />
client_addr | <br />
client_hostname | <br />
client_port | -1<br />
backend_start | 2012-07-02 15:02:51.146427+02<br />
xact_start | 2012-07-02 15:15:28.386865+02<br />
query_start | 2012-07-02 15:15:30.410834+02<br />
state_change | 2012-07-02 15:15:30.411287+02<br />
waiting | f<br />
state | idle in transaction<br />
query | DELETE FROM test;<br />
</pre><br />
<br />
With PostgreSQL 9.1, all we would have would be «idle in transaction».<br />
<br />
As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.<br />
The view pg_stat_replication has also changes. The column procpid is rename to pid, to also be consistent with other system views.<br />
<br />
==Change all SQL-level statistics timing values to float8-stored milliseconds <!-- (Tom Lane) -->==<br />
<br />
pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.<br />
<br />
==postgresql.conf parameters changes <!-- (Heikki Linnakangas, Tom Lane, Peter Eisentraut) -->==<br />
<br />
* silent_mode has been removed. Use pg_ctl -l postmaster.log<br />
* wal_sender_delay has been removed. It is no longer needed<br />
* custom_variable_classes has been removed. All «classes» are accepted without declaration now<br />
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files<br />
<br />
[[Category:PostgreSQL 9.2]]</div>Rjujuhttps://wiki.postgresql.org/index.php?title=What%27s_new_in_PostgreSQL_9.2&diff=17998What's new in PostgreSQL 9.22012-08-10T12:41:05Z<p>Rjuju: /* Ensure that xpath() escapes special characters in string values */</p>
<hr />
<div>{{Languages}}<br />
<br />
This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release &ndash; PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.<br />
<br />
'''This page is incomplete!'''<br />
<br />
=Major new features=<br />
<br />
==Index-only scans <!-- Robert Haas, Ibrar Ahmed, Heikki Linnakangas, Tom Lane -->==<br />
<br />
In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.<br />
<br />
It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.<br />
<br />
There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.<br />
<br />
The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).<br />
<br />
Here is an example.<br />
<br />
create table demo_ios (col1 float, col2 float, col3 text);<br />
<br />
In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains wont be that big in real life.<br />
<br />
insert into demo_ios select generate_series(1,100000000),random(), 'mynotsolongstring';<br />
<br />
select pg_size_pretty(pg_total_relation_size('demo_ios'));<br />
pg_size_pretty <br />
----------------<br />
6512 MB<br />
<br />
Let's pretend that the query is this:<br />
<br />
SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.02 AND 0.03<br />
<br />
In order to use an index only scan on this, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).<br />
<br />
CREATE index idx_demo_ios on demo_ios(col2,col1);<br />
<br />
We vacuum the visibility map to be up-to-date:<br />
<br />
VACUUM demo_ios;<br />
<br />
All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).<br />
<br />
Let's first try without Index Only Scans:<br />
<br />
set enable_indexonlyscan to off;<br />
<br />
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------------------<br />
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)<br />
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Rows Removed by Index Recheck: 68098621<br />
Buffers: shared hit=2 read=587779<br />
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Buffers: shared hit=2 read=3835<br />
Total runtime: 364390.127 ms<br />
<br />
<br />
With Index Only Scans:<br />
<br />
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------------------------------<br />
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)<br />
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))<br />
Heap Fetches: 0<br />
Buffers: shared hit=923073 read=3848<br />
Total runtime: 4297.405 ms<br />
<br />
<br />
<br />
As nothing is free, there are a few things to note:<br />
<br />
* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.<br />
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.<br />
* Gains will probably be smaller in real life situations.<br />
<br />
This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.<br />
<br />
==Replication improvements <!-- Fujii Masao, Simon Riggs, Magnus Hagander, Jun Ishizuka -->==<br />
<br />
Streaming Replication is getting even more polished with this release. One on the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources.<br />
<br />
Moreover, in case of a failover, it was very complicated to reconnect all the remaining slaves to the newly promoted master.<br />
<br />
To be on the safe side, it was easier to re-synchronize the slaves to the new masters from scratch, meaning that during this failover, only one server was active, and under heavy load, as it was used to rebuild all the slaves.<br />
<br />
* With 9.2, a slave can also be a replication master, allowing for cascading replication.<br />
<br />
Let's build this. We start with an already working 9.2 database.<br />
<br />
We set it up for replication:<br />
<br />
postgresql.conf:<br />
wal_level=hot_standby #(could be archive too)<br />
max_wal_senders=5<br />
hot_standby=on<br />
<br />
You'll probably also want to activate archiving in production, it won't be done here.<br />
<br />
pg_hba.conf (do not use trust in production):<br />
host replication replication_user 0.0.0.0/0 md5<br />
<br />
Create the user:<br />
create user replication_user replication password 'secret';<br />
<br />
Clone the database:<br />
<br />
pg_basebackup -h localhost -U replication_user -D data2<br />
Password:<br />
<br />
We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf):<br />
port=5433<br />
<br />
We add a recovery.conf to tell it how to stream from the master database:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret' <br />
<br />
pg_ctl -D data2 start<br />
server starting<br />
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9D0000B8<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, let's add a second slave, which will use this slave:<br />
<br />
<br />
pg_basebackup -h localhost -U replication_user -D data3 -p 5433<br />
Password: <br />
<br />
We edit data3's postgresql.conf to change the port:<br />
port=5434<br />
<br />
We modify the recovery.conf to stream from the slave:<br />
standby_mode = on<br />
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'<br />
<br />
We start the cluster:<br />
pg_ctl -D data3 start<br />
server starting<br />
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST<br />
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.<br />
LOG: creating missing WAL directory "pg_xlog/archive_status"<br />
LOG: entering standby mode<br />
LOG: streaming replication successfully connected to primary<br />
LOG: redo starts at 0/9D000020<br />
LOG: consistent recovery state reached at 0/9E000000<br />
LOG: database system is ready to accept read only connections<br />
<br />
Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).<br />
<br />
<br />
* As you may have noticed from the examble, pg_basebackup now works from slaves.<br />
<br />
* There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?<br />
<br />
pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:<br />
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user<br />
<br />
will connect to the master (or a slave), and start creating files: <br />
ls /tmp/new_logs/<br />
00000001000000000000009E.partial<br />
<br />
Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.<br />
<br />
* synchronous_commit has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, some people will be interested in this compromise.<br />
<br />
<br />
<br />
<br />
==JSON datatype==<br />
The JSON datatype is meant for storing JSON-structured data. (More info: [http://www.depesz.com/2012/02/12/waiting-for-9-2-json/ depesz blog])<br />
<br />
== Range Types ==<br />
[[RangeTypes]] are added.<br />
(More info: [http://www.depesz.com/2011/11/07/waiting-for-9-2-range-data-types/])<br />
<br />
=Performance improvements=<br />
<br />
This version has performance improvements on a very large range of domains (non-exaustive):<br />
<br />
* The most visible will probably be the Index Only Scans, which has already been introduced in this document.<br />
<br />
* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. <!-- Robert Haas --><br />
<br />
* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. <!-- Peter Geoghegan --><br />
<br />
* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption <!--Peter Geoghegan-->. This is especially useful on virtualized and embedded environments.<br />
<br />
* COPY has been improved, it will generate less WAL volume and less locks of tables's pages. <!-- Heikki Linnakangas --><br />
<br />
* Statistics are collected on array contents <!-- Alexander Korotkov -->, allowing for better estimations of selectivity on array operations.<br />
<br />
* The system can now track IO durations <!--Ants Aasma --><br />
<br />
This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.<br />
<br />
If you are interested in this timing data, it's better to first check if your system will support it without to much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:<br />
<br />
<pre><br />
$ pg_test_timing <br />
Testing timing overhead for 3 seconds.<br />
Per loop time including overhead: 28.02 nsec<br />
Histogram of timing durations:<br />
< usec: count percent<br />
32: 41 0.00004%<br />
16: 1405 0.00131%<br />
8: 200 0.00019%<br />
4: 388 0.00036%<br />
2: 2982558 2.78523%<br />
1: 104100166 97.21287%<br />
</pre><br />
<br />
Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].<br />
<br />
Anyway, here is the data you'll be able to collect if your system is ready for this:<br />
<br />
First, you'll get per-database statistics, which will now give accurate information about which database is doing most I/O:<br />
<br />
<pre><br />
=# select * from pg_stat_database where datname = 'mydb';<br />
-[ RECORD 1 ]--+------------------------------<br />
datid | 16384<br />
datname | mydb<br />
numbackends | 1<br />
xact_commit | 270<br />
xact_rollback | 2<br />
blks_read | 1961<br />
blks_hit | 17944<br />
tup_returned | 269035<br />
tup_fetched | 8850<br />
tup_inserted | 16<br />
tup_updated | 4<br />
tup_deleted | 45<br />
conflicts | 0<br />
temp_files | 0<br />
temp_bytes | 0<br />
deadlocks | 0<br />
blk_read_time | 583.774<br />
blk_write_time | 0<br />
stats_reset | 2012-07-03 17:18:54.796817+02<br />
</pre><br />
We see here that mydb has only consumed 583.774 milliseconds of read time.<br />
<br />
Explain will benefit from this too:<br />
<pre><br />
=# explain (analyze,buffers) select count(*) from mots ;<br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------<br />
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)<br />
Buffers: shared read=493<br />
I/O Timings: read=2.578<br />
Total runtime: 22.059 ms<br />
</pre><br />
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).<br />
<br />
And last, if you have enabled pg_stat_statements:<br />
<pre><br />
select * from pg_stat_statements where query ~ 'words';<br />
-[ RECORD 1 ]-------+---------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | select count(*) from words;<br />
calls | 2<br />
total_time | 78.332<br />
rows | 2<br />
shared_blks_hit | 0<br />
shared_blks_read | 986<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 58.427<br />
blk_write_time | 0<br />
</pre><br />
<br />
* As for every version, the optimizer has received its share of improvements <!-- Tom Lane--><br />
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.<br />
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it wanted to.<br />
<br />
This example is straight from the developpers mailing lists <!-- Andres Freund -->:<br />
<br />
<pre><br />
CREATE TABLE a (<br />
a_id serial PRIMARY KEY NOT NULL,<br />
b_id integer<br />
);<br />
CREATE INDEX a__b_id ON a USING btree (b_id);<br />
<br />
<br />
CREATE TABLE b (<br />
b_id serial NOT NULL,<br />
c_id integer<br />
);<br />
CREATE INDEX b__c_id ON b USING btree (c_id);<br />
<br />
<br />
CREATE TABLE c (<br />
c_id serial PRIMARY KEY NOT NULL,<br />
value integer UNIQUE<br />
);<br />
<br />
INSERT INTO b (b_id, c_id)<br />
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO a(b_id)<br />
SELECT g.i FROM generate_series(1, 50000) g(i);<br />
<br />
INSERT INTO c(c_id,value)<br />
VALUES (1,1);<br />
</pre><br />
<br />
So we have a referencing b, b referencing c.<br />
<br />
Here is an example of a query working badly with PostgreSQL 9.1:<br />
<br />
<pre><br />
EXPLAIN ANALYZE SELECT 1 <br />
FROM <br />
c<br />
WHERE<br />
EXISTS (<br />
SELECT * <br />
FROM a<br />
JOIN b USING (b_id)<br />
WHERE b.c_id = c.c_id)<br />
AND c.value = 1;<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)<br />
Join Filter: (c.c_id = b.c_id)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)<br />
Hash Cond: (a.b_id = b.b_id)<br />
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)<br />
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)<br />
Buckets: 8192 Batches: 1 Memory Usage: 1954kB<br />
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)<br />
Total runtime: 13.842 ms<br />
</pre><br />
<br />
Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.<br />
<br />
Here's what 9.2 does with this query:<br />
<br />
<pre><br />
QUERY PLAN <br />
----------------------------------------------------------------------------------------------------------------------------<br />
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)<br />
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)<br />
Index Cond: (value = 1)<br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
The «parameterized path» is:<br />
<pre><br />
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)<br />
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)<br />
Index Cond: (c_id = c.c_id)<br />
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)<br />
Index Cond: (b_id = b.b_id)<br />
Total runtime: 0.089 ms<br />
</pre><br />
<br />
This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.<br />
<br />
This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.<br />
<br />
<br />
=SP-GIST=<br />
TODO<br />
<br />
=pg_stat_statements=<br />
<br />
This contrib module has received a lot of improvements in this version:<br />
<br />
* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.<br />
<br />
<pre><br />
=#select * from words where word= 'foo';<br />
word <br />
------<br />
(0 ligne)<br />
<br />
=# select * from words where word= 'bar';<br />
word <br />
------<br />
bar<br />
<br />
=#select * from pg_stat_statements where query like '%words where%';<br />
-[ RECORD 1 ]-------+-----------------------------------<br />
userid | 10<br />
dbid | 16384<br />
query | select * from words where word= ?;<br />
calls | 2<br />
total_time | 142.314<br />
rows | 1<br />
shared_blks_hit | 3<br />
shared_blks_read | 5<br />
shared_blks_dirtied | 0<br />
shared_blks_written | 0<br />
local_blks_hit | 0<br />
local_blks_read | 0<br />
local_blks_dirtied | 0<br />
local_blks_written | 0<br />
temp_blks_read | 0<br />
temp_blks_written | 0<br />
blk_read_time | 142.165<br />
blk_write_time | 0<br />
<br />
</pre><br />
<br />
The two queries are shown as one in pg_stat_statements.<br />
<br />
* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That way it is easier to use, and avoids the double-counting there was with PostgreSQL 9.1.<br />
<br />
* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.<br />
<br />
= Explain improvements=<br />
<br />
* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive <!--Tomas Vondra--><br />
<br />
<br />
Have EXPLAIN ANALYZE report the number of rows rejected by filter steps (Marko Tiikkaja)<br />
<br />
=Backward compatibility=<br />
<br />
These changes may incur regressions in your applications.<br />
<br />
==Ensure that xpath() escapes special characters in string values <!-- (Florian Pflug)--> ==<br />
<br />
Before 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
<<br />
<br />
'<' Isn't valid XML.<br />
</pre><br />
With 9.2:<br />
<pre><br />
SELECT (XPATH('/*/text()', '<root>&amp;lt;</root>'))[1];<br />
xpath <br />
-------<br />
&amp;lt;<br />
</pre><br />
<br />
==Remove hstore's => operator <!-- (Robert Haas)-->==<br />
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.<br />
<br />
In 9.1:<br />
<pre><br />
SELECT pg_typeof('a'=>'b');<br />
pg_typeof <br />
-----------<br />
hstore<br />
(1 row)<br />
<br />
=# SELECT 'a'=>'b';<br />
?column? <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# SELECT pg_typeof('a'=>'b');<br />
pg_typeof <br />
-----------<br />
hstore<br />
(1 row)<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown at character 11<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
STATEMENT: SELECT 'a'=>'b';<br />
ERROR: operator does not exist: unknown => unknown<br />
LINE 1: SELECT 'a'=>'b';<br />
^<br />
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.<br />
</pre><br />
<br />
It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:<br />
<br />
<pre><br />
=# select hstore('a=>b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
<br />
=# select hstore('a','b');<br />
hstore <br />
----------<br />
"a"=>"b"<br />
(1 row)<br />
</pre><br />
are still two valid ways to input a hstore.<br />
<br />
"=>" is removed as an operator as it is a reserved keyword in SQL.<br />
<br />
<br />
==Have pg_relation_size() and friends return NULL if the object does not exist <!-- (Phil Sorber)-->==<br />
<br />
A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.<br />
<br />
<br />
==Remove the spclocation field from pg_tablespace <!-- (Magnus Hagander)-->==<br />
<br />
The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.<br />
<br />
So this field has been removed. To get the tablespace's location, use pg_tablespace_location():<br />
<br />
<pre><br />
=# select *, pg_tablespace_location(oid) as spclocation from pg_tablespace;<br />
spcname | spcowner | spcacl | spcoptions | spclocation <br />
------------+----------+--------+------------+----------------<br />
pg_default | 10 | | | <br />
pg_global | 10 | | | <br />
tmptblspc | 10 | | | /tmp/tmptblspc<br />
</pre><br />
<br />
==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight <!-- (Tom Lane) -->==<br />
<br />
<br />
With PostgreSQL 9.1:<br />
<br />
<pre><br />
=#SELECT extract(epoch from '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
<br />
=# SELECT extract(epoch from '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
<br />
There is no difference in behaviour between a timstamp with or without timezone.<br />
<br />
With 9.1:<br />
<pre><br />
=#SELECT extract(epoch from '2012-07-02 00:00:00'::timestamp);<br />
date_part <br />
------------<br />
1341187200<br />
(1 row)<br />
<br />
=# SELECT extract(epoch from '2012-07-02 00:00:00'::timestamptz);<br />
date_part <br />
------------<br />
1341180000<br />
(1 row)<br />
</pre><br />
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.<br />
<br />
<br />
==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 <!-- (Bruce Momjian)-->==<br />
<br />
The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.<br />
<br />
Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.<br />
<br />
With 9.1:<br />
<pre><br />
=# SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
1200-07-02<br />
</pre><br />
<br />
With 9.2:<br />
<pre><br />
SELECT to_date('200-07-02','YYY-MM-DD');<br />
to_date <br />
------------<br />
2200-07-02<br />
</pre><br />
<br />
==pg_stat_activity's definition has changed <!--Magnus Hagander -->==<br />
<br />
The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:<br />
<br />
* current_query disappears and is replaced by two columns:<br />
** state: is the session running a query, waiting<br />
** query: what is the last run (or still running) query<br />
* The column procpid is renamed to pid, to be consistent with other system views<br />
<br />
The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.<br />
<br />
Here is an example:<br />
<pre><br />
-[ RECORD 1 ]----+---------------------------------<br />
datid | 16384<br />
datname | postgres<br />
pid | 20804<br />
usesysid | 10<br />
usename | postgres<br />
application_name | psql<br />
client_addr | <br />
client_hostname | <br />
client_port | -1<br />
backend_start | 2012-07-02 15:02:51.146427+02<br />
xact_start | 2012-07-02 15:15:28.386865+02<br />
query_start | 2012-07-02 15:15:30.410834+02<br />
state_change | 2012-07-02 15:15:30.411287+02<br />
waiting | f<br />
state | idle in transaction<br />
query | DELETE FROM test;<br />
</pre><br />
<br />
With PostgreSQL 9.1, all we would have would be «idle in transaction».<br />
<br />
As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.<br />
<br />
==Change all SQL-level statistics timing values to float8-stored milliseconds <!-- (Tom Lane) -->==<br />
<br />
pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.<br />
<br />
==postgresql.conf parameters changes <!-- (Heikki Linnakangas, Tom Lane, Peter Eisentraut) -->==<br />
<br />
* silent_mode has been removed. Use pg_ctl -l postmaster.log<br />
* wal_sender_delay has been removed. It is no longer needed<br />
* custom_variable_classes has been removed. All «classes» are accepted without declaration now<br />
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files<br />
<br />
[[Category:PostgreSQL 9.2]]</div>Rjuju