PostgreSQL wiki - User contributions [en]

MVCC violations

2014-11-03T16:56:50Z

Intgr: Document violations using STABLE PL/pgSQL functions; fix table layouts

In general, PostgreSQL aims to be strict with ACID transaction semantics. But there are some documented cases that, for performance or usability reasons, violate the MVCC protocol, and thus the atomicity, consistency or isolation properties of ACID transactions. Such instances are sometimes referred to as "MVCC violations". This page aims to illustrate when such situations occur.

In general, MVCC violations are detectable using:
* REPEATABLE READ isolation level -- usually the easiest way to reliably reveal violations.
* With the default READ COMMITTED level, violations are visible only rarely when winning a tight race condition, or when a 3rd session happens to synchronize the backends in a certain way.
* Within STABLE PL/pgSQL functions, even when using the READ COMMITTED level, due to snapshot allocation optimizations with STABLE functions.

== TRUNCATE TABLE ==

Using TRUNCATE is a common technique for quickly replacing the contents of a table with new contents from an external source. Due to performance tradeoffs, the table can temporarily appear empty to concurrent transactions.

To avoid this problem, use the <code>DELETE FROM</code> statement, which has a higher performance and I/O cost.

=== With REPEATABLE READ isolation level ===

This issue is the easiest do demonstrate in the REPEATABLE READ isolation level. Any reading transaction should see either "old data" or "new data" in the table, never an empty table.

Test case:
{| class="wikitable"
!colspan=2|Preparation
|-
|colspan=2|
<source lang="sql">
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">SELECT * FROM a;
text
------
(0 rows)
</source>
|
|}

=== With READ COMMITTED (default) isolation ===

This issue is still present in the default READ COMMITTED isolation level, but more difficult to trigger reliably, because a fresh snapshot is allocated for each new query.

Thanks to Andrew Gierth for pointing out this scenario and Andres Freund for coming up with this test case.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE sync(text text);
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Synchronizing session
! Reading session
! Writing session
|-
|
| <source lang="sql">
PREPARE prep AS
SELECT * FROM sync UNION ALL SELECT * FROM a;
</source>
|
|-
| <source lang="sql">
BEGIN;
LOCK TABLE sync;
</source>
|
|
|-
|
| <source lang="sql">
EXECUTE prep;
-- blocks behind session #1
</source>
|
|-
|
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">ROLLBACK;</source>
| <source lang="sql">
-- session unblocks after #1 rollback
text
------
(0 rows)
</source>
|}

=== STABLE functions ===

MVCC violations are also easily visible to STABLE PL/pgSQL functions due to the way snapshots are allocated. In earlier versions (9.1 and earlier. Not sure about 9.2???), this also affected SQL language functions. [http://www.postgresql.org/docs/current/static/xfunc-volatility.html Documentation for function snapshot allocation]. Thanks to Marko Tiikkaja for the test case.

Test case:
{| class="wikitable"
!colspan=2|Preparation
|-
|colspan=2|
<source lang="sql">
CREATE TABLE a AS SELECT 'old data'::text;
CREATE FUNCTION get_a()
RETURNS SETOF text LANGUAGE plpgsql STABLE
AS 'BEGIN RETURN QUERY SELECT text FROM a; END';
</source>
|-
! Writing session
! Reading session
|-
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
</source>
|
|-
|
| <source lang="sql">
SELECT * FROM get_a();
-- blocks behind session #1
</source>
|-
| <source lang="sql">COMMIT;</source>
|
|-
|
| <source lang="sql">
-- session unblocks after #1 commits
get_a
-------
(0 rows)
</source>
|}

== COPY FREEZE ==

The FREEZE option of the COPY command was introduced in PostgreSQL version 9.3 to avoid expensive vacuum maintenance later on the restored table. In this case, the reading session should see an empty table, since its snapshot started before any rows were inserted (copied).

To avoid this problem, simply don't use the FREEZE option.

Test case:
{| class="wikitable"
!colspan=2|Preparation
|-
|colspan=2|
<source lang="sql">
CREATE TABLE a (text text);
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
COPY a FROM STDIN WITH (FREEZE);
phantom row
\.
COMMIT;</source>
|-
| <source lang="sql">
SELECT * FROM a;
text
-------------
phantom row
(1 row)
</source>
|
|}

== UPDATE & SELECT FOR UPDATE ==

In the READ COMMITTED isolation level, any command that acquires row locks, when faced with concurrently modified rows, is able to "reach into the future" and see the latest committed row version outside of its own snapshot. Arguably it doesn't violate ACID properties because READ COMMITTED is allowed to see concurrently committed rows. But it goes against usual PostgreSQL MVCC behavior where a single statement cannot see any anomalies even in READ COMMITTED.

To avoid this, simply use a higher isolation level. Unlike previous examples, this one cannot happen in REPEATABLE READ and higher levels, instead it throws the error "could not serialize access due to concurrent update".

User-visible symptoms include:
* Result rows violating ordering specified by ORDER BY clause because the ordering is applied in the original snapshot ''before'' acquiring row locks.
* Result rows that appear to violate causality (and possibly FOREIGN KEY constraints?)
* WHERE clause expressions evaulated twice for the same row -- once for the old copy and again for the new copy.
* Result containing fewer rows because the new locked version no longer matches the WHERE clause. (Note: the inverse is not true: rows whose old version didn't match will never appear in the result set, even if their new version does).

Test case:
{| class="wikitable"
!colspan=2|Preparation
|-
|colspan=2|
<source lang="sql">
CREATE TABLE a AS SELECT 'a'::text UNION ALL SELECT 'b';
</source>
|-
! Writing session
! Reading session
|-
| <source lang="sql">
BEGIN;
UPDATE a SET text='zzz' WHERE text='a';
</source>
|
|-
|
| <source lang="sql">
SELECT * FROM a ORDER BY text FOR UPDATE;
-- blocks behind session #1
</source>
|-
| <source lang="sql">
COMMIT;
</source>
|
|-
|
| <source lang="sql">
-- session unblocks after #1 commits
text
------
zzz -- notice rows are out of order
b
(2 rows)
</source>
|}

== ALTER TABLE rewrite ==

When querying a table that has been rewritten by ALTER TABLE in another session, the table may appear temporarily empty to concurrent transactions. Again, this is easier to reproduce using the REPEATABLE READ isolation level. Thanks to Marko Tiikkaja for the test case.

Test case:
{| class="wikitable"
!colspan=2|Preparation
|-
|colspan=2|
<source lang="sql">
CREATE TABLE a AS SELECT 'data'::text;
</source>
|-
! Writing session
! Reading session
|-
| <source lang="sql">
BEGIN;
ALTER TABLE a
ADD COLUMN b int DEFAULT 0;
</source>
|
|-
|
| <source lang="sql">
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT * FROM a;
-- blocks behind session #1
</source>
|-
| <source lang="sql">COMMIT;</source>
|
|-
|
| <source lang="sql">
-- session unblocks after #1 commits
text | b
------+---
(0 rows)
</source>
|}

== Historic issues ==

* In PostgreSQL 8.2 and earlier, the CLUSTER command could cause DELETEd/UPDATEd rows to disappear for older snapshots.

MVCC violations

2014-11-03T16:03:09Z

Intgr: ALTER TABLE rewrite

In general, PostgreSQL aims to be strict with ACID transaction semantics. But there are some documented cases that, for performance or usability reasons, violate the MVCC protocol, and thus the atomicity, consistency or isolation properties of ACID transactions. Such instances are sometimes referred to as "MVCC violations". This page aims to illustrate when such situations occur.

In general, such scenarios are easy to demonstrate reliably with the REPEATABLE READ isolation level, but occur rarely and unpredictably in the default READ COMMITTED level.

== TRUNCATE TABLE ==

Using TRUNCATE is a common technique for quickly replacing the contents of a table with new contents from an external source. Due to performance tradeoffs, the table can temporarily appear empty to concurrent transactions.

To avoid this problem, use the <code>DELETE FROM</code> statement, which has a higher performance and I/O cost.

=== With REPEATABLE READ isolation level ===

This issue is the easiest do demonstrate in the REPEATABLE READ isolation level. Any reading transaction should see either "old data" or "new data" in the table, never an empty table.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">SELECT * FROM a;
text
------
(0 rows)
</source>
|}

=== With READ COMMITTED (default) isolation ===

This issue is still present in the default READ COMMITTED isolation level, but more difficult to trigger reliably, because a fresh snapshot is allocated for each new query.

Thanks to Andrew Gierth for pointing out this scenario and Andres Freund for coming up with this test case.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE sync(text text);
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Synchronizing session
! Reading session
! Writing session
|-
|
| <source lang="sql">
PREPARE prep AS
SELECT * FROM sync UNION ALL SELECT * FROM a;
</source>
|
|-
| <source lang="sql">
BEGIN;
LOCK TABLE sync;
</source>
|
|
|-
|
| <source lang="sql">
EXECUTE prep;
-- blocks behind session #1
</source>
|
|-
|
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">ROLLBACK;</source>
| <source lang="sql">
-- session unblocks after #1 rollback
text
------
(0 rows)
</source>
|}

== COPY FREEZE ==

The FREEZE option of the COPY command was introduced in PostgreSQL version 9.3 to avoid expensive vacuum maintenance later on the restored table. In this case, the reading session should see an empty table, since its snapshot started before any rows were inserted (copied).

To avoid this problem, simply don't use the FREEZE option.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a (text text);
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
COPY a FROM STDIN WITH (FREEZE);
phantom row
\.
COMMIT;</source>
|-
| <source lang="sql">
SELECT * FROM a;
text
-------------
phantom row
(1 row)
</source>
|}

== UPDATE & SELECT FOR UPDATE ==

In the READ COMMITTED isolation level, any command that acquires row locks, when faced with concurrently modified rows, is able to "reach into the future" and see the latest committed row version outside of its own snapshot. Arguably it doesn't violate ACID properties because READ COMMITTED is allowed to see concurrently committed rows. But it goes against usual PostgreSQL MVCC behavior where a single statement cannot see any anomalies even in READ COMMITTED.

To avoid this, simply use a higher isolation level. Unlike previous examples, this one cannot happen in REPEATABLE READ and higher levels, instead it throws the error "could not serialize access due to concurrent update".

User-visible symptoms include:
* Result rows violating ordering specified by ORDER BY clause because the ordering is applied in the original snapshot ''before'' acquiring row locks.
* Result rows that appear to violate causality (and possibly FOREIGN KEY constraints?)
* WHERE clause expressions evaulated twice for the same row -- once for the old copy and again for the new copy.
* Result containing fewer rows because the new locked version no longer matches the WHERE clause. (Note: the inverse is not true: rows whose old version didn't match will never appear in the result set, even if their new version does).

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'a'::text UNION ALL SELECT 'b';
</source>
|-
! Writing session
! Reading session
|-
| <source lang="sql">
BEGIN;
UPDATE a SET text='zzz' WHERE text='a';
</source>
|
|-
|
| <source lang="sql">
SELECT * FROM a ORDER BY text FOR UPDATE;
-- blocks behind session #1
</source>
|-
| <source lang="sql">
COMMIT;
</source>
|
|-
|
| <source lang="sql">
-- session unblocks after #1 commits
text
------
zzz -- notice rows are out of order
b
(2 rows)
</source>
|}

== ALTER TABLE rewrite ==

Querying a table that has been rewritten by ALTER TABLE in another session, the table may appear temporarily empty to concurrent transactions. Again, this is easier to reproduce using the REPEATABLE READ isolation level. Thanks to Marko Tiikkaja for the test case.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'data'::text;
</source>
|-
! Writing session
! Reading session
|-
| <source lang="sql">
BEGIN;
ALTER TABLE a
ADD COLUMN b int DEFAULT 0;
</source>
|
|-
|
| <source lang="sql">
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT * FROM a;
-- blocks behind session #1
</source>
|-
| <source lang="sql">COMMIT;</source>
|
|-
|
| <source lang="sql">
-- session unblocks after #1 commits
text | b
------+---
(0 rows)
</source>
|}

== Historic issues ==

* In PostgreSQL 8.2 and earlier, the CLUSTER command could cause DELETEd/UPDATEd rows to disappear for older snapshots.

MVCC violations

2014-08-04T09:03:01Z

Intgr: /* UPDATE & SELECT FOR UPDATE */ clarification

In general, PostgreSQL aims to be strict with ACID transaction semantics. But there are some documented cases that, for performance or usability reasons, violate the MVCC protocol, and thus the atomicity, consistency or isolation properties of ACID transactions. Such instances are sometimes referred to as "MVCC violations". This page aims to illustrate when such situations occur.

In general, such scenarios are easy to demonstrate reliably with the REPEATABLE READ isolation level, but occur rarely and unpredictably in the default READ COMMITTED level.

== TRUNCATE TABLE ==

Using TRUNCATE is a common technique for quickly replacing the contents of a table with new contents from an external source. Due to performance tradeoffs, the table can temporarily appear empty to concurrent transactions.

To avoid this problem, use the <code>DELETE FROM</code> statement, which has a higher performance and I/O cost.

=== With REPEATABLE READ isolation level ===

This issue is the easiest do demonstrate in the REPEATABLE READ isolation level. Any reading transaction should see either "old data" or "new data" in the table, never an empty table.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">SELECT * FROM a;
text
------
(0 rows)
</source>
|}

=== With READ COMMITTED (default) isolation ===

This issue is still present in the default READ COMMITTED isolation level, but more difficult to trigger reliably, because a fresh snapshot is allocated for each new query.

Thanks to Andrew Gierth for pointing out this scenario and Andres Freund for coming up with this test case.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE sync(text text);
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Synchronizing session
! Reading session
! Writing session
|-
|
| <source lang="sql">
PREPARE prep AS
SELECT * FROM sync UNION ALL SELECT * FROM a;
</source>
|
|-
| <source lang="sql">
BEGIN;
LOCK TABLE sync;
</source>
|
|
|-
|
| <source lang="sql">
EXECUTE prep;
-- blocks behind session #1
</source>
|
|-
|
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">ROLLBACK;</source>
| <source lang="sql">
-- session unblocks after #1 rollback
text
------
(0 rows)
</source>
|}

== COPY FREEZE ==

The FREEZE option of the COPY command was introduced in PostgreSQL version 9.3 to avoid expensive vacuum maintenance later on the restored table. In this case, the reading session should see an empty table, since its snapshot started before any rows were inserted (copied).

To avoid this problem, simply don't use the FREEZE option.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a (text text);
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
COPY a FROM STDIN WITH (FREEZE);
phantom row
\.
COMMIT;</source>
|-
| <source lang="sql">
SELECT * FROM a;
text
-------------
phantom row
(1 row)
</source>
|}

== UPDATE & SELECT FOR UPDATE ==

In the READ COMMITTED isolation level, any command that acquires row locks, when faced with concurrently modified rows, is able to "reach into the future" and see the latest committed row version outside of its own snapshot. Arguably it doesn't violate ACID properties because READ COMMITTED is allowed to see concurrently committed rows. But it goes against usual PostgreSQL MVCC behavior where a single statement cannot see any anomalies even in READ COMMITTED.

To avoid this, simply use a higher isolation level. Unlike previous examples, this one cannot happen in REPEATABLE READ and higher levels, instead it throws the error "could not serialize access due to concurrent update".

User-visible symptoms include:
* Result rows violating ordering specified by ORDER BY clause because the ordering is applied in the original snapshot ''before'' acquiring row locks.
* Result rows that appear to violate causality (and possibly FOREIGN KEY constraints?)
* WHERE clause expressions evaulated twice for the same row -- once for the old copy and again for the new copy.
* Result containing fewer rows because the new locked version no longer matches the WHERE clause. (Note: the inverse is not true: rows whose old version didn't match will never appear in the result set, even if their new version does).

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'a'::text UNION ALL SELECT 'b';
</source>
|-
! Writing session
! Reading session
|-
| <source lang="sql">
BEGIN;
UPDATE a SET text='zzz' WHERE text='a';
</source>
|
|-
|
| <source lang="sql">
SELECT * FROM a ORDER BY text FOR UPDATE;
-- blocks behind session #1
</source>
|-
| <source lang="sql">
COMMIT;
</source>
|
|-
|
| <source lang="sql">
-- session unblocks after #1 commits
text
------
zzz -- notice rows are out of order
b
(2 rows)
</source>
|}

== Historic issues ==

* In PostgreSQL 8.2 and earlier, the CLUSTER command could cause DELETEd/UPDATEd rows to disappear for older snapshots.

MVCC violations

2014-08-04T08:42:36Z

Intgr: UPDATE & SELECT FOR UPDATE

In general, PostgreSQL aims to be strict with ACID transaction semantics. But there are some documented cases that, for performance or usability reasons, violate the MVCC protocol, and thus the atomicity, consistency or isolation properties of ACID transactions. Such instances are sometimes referred to as "MVCC violations". This page aims to illustrate when such situations occur.

In general, such scenarios are easy to demonstrate reliably with the REPEATABLE READ isolation level, but occur rarely and unpredictably in the default READ COMMITTED level.

== TRUNCATE TABLE ==

Using TRUNCATE is a common technique for quickly replacing the contents of a table with new contents from an external source. Due to performance tradeoffs, the table can temporarily appear empty to concurrent transactions.

To avoid this problem, use the <code>DELETE FROM</code> statement, which has a higher performance and I/O cost.

=== With REPEATABLE READ isolation level ===

This issue is the easiest do demonstrate in the REPEATABLE READ isolation level. Any reading transaction should see either "old data" or "new data" in the table, never an empty table.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">SELECT * FROM a;
text
------
(0 rows)
</source>
|}

=== With READ COMMITTED (default) isolation ===

This issue is still present in the default READ COMMITTED isolation level, but more difficult to trigger reliably, because a fresh snapshot is allocated for each new query.

Thanks to Andrew Gierth for pointing out this scenario and Andres Freund for coming up with this test case.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE sync(text text);
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Synchronizing session
! Reading session
! Writing session
|-
|
| <source lang="sql">
PREPARE prep AS
SELECT * FROM sync UNION ALL SELECT * FROM a;
</source>
|
|-
| <source lang="sql">
BEGIN;
LOCK TABLE sync;
</source>
|
|
|-
|
| <source lang="sql">
EXECUTE prep;
-- blocks behind session #1
</source>
|
|-
|
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">ROLLBACK;</source>
| <source lang="sql">
-- session unblocks after #1 rollback
text
------
(0 rows)
</source>
|}

== COPY FREEZE ==

The FREEZE option of the COPY command was introduced in PostgreSQL version 9.3 to avoid expensive vacuum maintenance later on the restored table. In this case, the reading session should see an empty table, since its snapshot started before any rows were inserted (copied).

To avoid this problem, simply don't use the FREEZE option.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a (text text);
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
COPY a FROM STDIN WITH (FREEZE);
phantom row
\.
COMMIT;</source>
|-
| <source lang="sql">
SELECT * FROM a;
text
-------------
phantom row
(1 row)
</source>
|}

== UPDATE & SELECT FOR UPDATE ==

In the READ COMMITTED isolation level, any command that acquires row locks, when faced with concurrently modified rows, is able to "reach into the future" and see the latest committed row version outside of its own snapshot. Arguably it doesn't violate ACID properties because READ COMMITTED is allowed to see concurrently committed rows. But it goes against usual PostgreSQL MVCC behavior where a single statement cannot see any anomalies even in READ COMMITTED.

To avoid this, simply use a higher isolation level. Unlike previous examples, this one cannot happen in REPEATABLE READ and higher levels, instead it throws the error "could not serialize access due to concurrent update".

User-visible symptoms include:
* Result rows violating ordering specified by ORDER BY clause because the ordering is applied in the original snapshot ''before'' acquiring row locks.
* Result rows that appear to violate causality (and possibly FOREIGN KEY constraints?)
* WHERE clause expressions evaulated twice for the same row -- once for the old copy and again for the new copy.
* Result containing fewer rows because the new locked version no longer matches the WHERE clause.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'a'::text UNION ALL SELECT 'b';
</source>
|-
! Writing session
! Reading session
|-
| <source lang="sql">
BEGIN;
UPDATE a SET text='zzz' WHERE text='a';
</source>
|
|-
|
| <source lang="sql">
SELECT * FROM a ORDER BY text FOR UPDATE;
-- blocks behind session #1
</source>
|-
| <source lang="sql">
COMMIT;
</source>
|
|-
|
| <source lang="sql">
-- session unblocks after #1 commits
text
------
zzz -- notice rows are out of order
b
(2 rows)
</source>
|}

== Historic issues ==

* In PostgreSQL 8.2 and earlier, the CLUSTER command could cause DELETEd/UPDATEd rows to disappear for older snapshots.

MVCC violations

2014-08-01T14:38:02Z

Intgr: CLUSTER in PostgreSQL 8.2 and earlier

In general, PostgreSQL aims to be strict with ACID transaction semantics. But there are some documented cases that, for performance reasons, violate the MVCC protocol, and thus the atomicity, consistency or isolation properties of ACID transactions. Such instances are sometimes referred to as "MVCC violations". This page aims to illustrate when such situations occur.

In general, such scenarios are easy to demonstrate reliably with the REPEATABLE READ isolation level, but occur rarely and unpredictably in the default READ COMMITTED level.

== TRUNCATE TABLE ==

Using TRUNCATE is a common technique for quickly replacing the contents of a table with new contents from an external source. Due to performance tradeoffs, the table can temporarily appear empty to concurrent transactions.

To avoid this problem, use the <code>DELETE FROM</code> statement, which has a higher performance and I/O cost.

=== With REPEATABLE READ isolation level ===

This issue is the easiest do demonstrate in the REPEATABLE READ isolation level. Any reading transaction should see either "old data" or "new data" in the table, never an empty table.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">SELECT * FROM a;
text
------
(0 rows)
</source>
|}

=== With READ COMMITTED (default) isolation ===

This issue is still present in the default READ COMMITTED isolation level, but more difficult to trigger reliably, because a fresh snapshot is allocated for each new query.

Thanks to Andrew Gierth for pointing out this scenario and Andres Freund for coming up with this test case.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE sync(text text);
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Synchronizing session
! Reading session
! Writing session
|-
|
| <source lang="sql">
PREPARE prep AS
SELECT * FROM sync UNION ALL SELECT * FROM a;
</source>
|
|-
| <source lang="sql">
BEGIN;
LOCK TABLE sync;
</source>
|
|
|-
|
| <source lang="sql">
EXECUTE prep;
-- blocks behind session #1
</source>
|
|-
|
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">ROLLBACK;</source>
| <source lang="sql">
-- session unblocks after #1 rollback
text
------
(0 rows)
</source>
|}

== COPY FREEZE ==

The FREEZE option of the COPY command was introduced in PostgreSQL version 9.3 to avoid expensive vacuum maintenance later on the restored table. In this case, the reading session should see an empty table, since its snapshot started before any rows were inserted (copied).

To avoid this problem, simply don't use the FREEZE option.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a (text text);
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
COPY a FROM STDIN WITH (FREEZE);
phantom row
\.
COMMIT;</source>
|-
| <source lang="sql">
SELECT * FROM a;
text
-------------
phantom row
(1 row)
</source>
|}

== Historic issues ==

* In PostgreSQL 8.2 and earlier, the CLUSTER command could cause DELETEd/UPDATEd rows to disappear for older snapshots.

MVCC violations

2014-08-01T14:24:24Z

Intgr: Add COPY FREEZE test case

MVCC violations

2014-08-01T14:09:16Z

Intgr: Test cases

In general, PostgreSQL aims to be strict with ACID transaction semantics. But there are some documented cases that, for performance reasons, violate the MVCC protocol, and thus the atomicity, consistency or isolation properties of ACID transactions. Such instances are sometimes referred to as "MVCC violations". This page aims to illustrate when such situations occur.

In general, such scenarios are easy to demonstrate reliably with the REPEATABLE READ isolation level, but occur rarely and unpredictably in the default READ COMMITTED level.

== TRUNCATE TABLE ==

Using TRUNCATE is a common technique for quickly replacing the contents of a table with new content. Due to performance tradeoffs, the table can temporarily appear empty to concurrent transactions.

To avoid this problem, use the <code>DELETE FROM</code> statement, which has a higher performance and I/O cost.

=== With REPEATABLE READ isolation level ===

This issue is the easiest do demonstrate in the REPEATABLE READ isolation level. Any reading transaction should see either "old data" or "new data" in the table, never an empty table.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Reading session
! Writing session
|-
| <source lang="sql">BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT txid_current(); -- force snapshot allocation
</source>
|
|-
|
| <source lang="sql">BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">SELECT * FROM a;
text
------
(0 rows)
</source>
|}

=== With READ COMMITTED (default) isolation ===

This issue is still present in the default READ COMMITTED isolation level, but more difficult to trigger reliably, because a fresh snapshot is allocated for each new query.

Thanks to Andrew Gierth for pointing out this scenario and Andres Freund for coming up with this test case.

Test case:
{| class="wikitable"
!colspan=3|Preparation
|-
|colspan=3|
<source lang="sql">
CREATE TABLE sync(text text);
CREATE TABLE a AS SELECT 'old data'::text;
</source>
|-
! Synchronizing session
! Reading session
! Writing session
|-
|
| <source lang="sql">
PREPARE prep AS
SELECT * FROM sync UNION ALL SELECT * FROM a;
</source>
|
|-
| <source lang="sql">
BEGIN;
LOCK TABLE sync;
</source>
|
|
|-
|
| <source lang="sql">
EXECUTE prep;
-- blocks behind session #1
</source>
|
|-
|
|
| <source lang="sql">
BEGIN;
TRUNCATE TABLE a;
INSERT INTO a VALUES ('new data');
COMMIT;</source>
|-
| <source lang="sql">ROLLBACK;</source>
| <source lang="sql">
-- session unblocks after #1 rollback
text
------
(0 rows)
</source>
|}

MVCC violations

2014-08-01T13:35:32Z

Intgr: Created page with "In general, PostgreSQL aims to be strict with ACID transaction semantics. But there are some documented cases that, for performance reasons, violate the MVCC protocol, and thu..."

What's new in PostgreSQL 9.4

2014-05-22T17:24:14Z

Intgr: correct plural

This page contains an overview of PostgreSQL Version 9.4's features, including descriptions, testing and usage information, and links to blog posts containing further information. See also [[PostgreSQL 9.4 Open Items]].

=Major new features=

==Replication improvements==

===Replication slots ===

TODO: Blog posts, redundancy of wal_keep_segments et al.

Replication slots allow standbys to provide information to the primary or upstream cascading standby as to the point they've reached in the write-ahead log. This information is available to the primary even when the standby is offline or disconnected. This eliminates the need for wal_keep_segments, which required the admin to estimate how many extra WAL files would be needed to be kept around in order to ensure supply to the standby in case of excessive replication lag.

Here is an example:

On the primary server, set max_replication_slots to allow the standby to register to a slot:

max_replication_slots = 1

The wal_level will also need to be set to at least "archive", but as this example will be using a hot standby, we'll set it to "hot_standby":

wal_level = hot_standby

Restart the primary for these settings to take effect, then connect to the primary and create a replication slot:

SELECT * FROM pg_create_physical_replication_slot('standby_replication_slot');

A standby can then use this replication slot by specifying it as the primary_slotname parameter in its recovery.conf:

primary_slotname = 'standby_replication_slot'

On the primary, you can see this being used:

postgres=# SELECT * FROM pg_replication_slots;
slot_name | plugin | slot_type | datoid | database | active | xmin | catalog_xmin | restart_lsn
-----------------------+--------+-----------+--------+----------+--------+------+--------------+-------------
standby_physical_slot | | physical | | | t | | | 0/3000420
(1 row)

Now the primary will keep hold of WAL for as along as it needs to, which is until all standbys have reported that they have progressed beyond that point. Note, however, that if a standby goes offline, and the replication slow isn't dropped, WAL will build up on the primary. So if a standby needs to be decommissioned, it's slot should then be dropped like so:

# SELECT pg_drop_replication_slot('standby_replication_slot');
pg_drop_replication_slot
--------------------------

(1 row)

Otherwise, ensure there is enough disk capacity available to keep WAL on the primary until the standby returns.

In future logical replication will also take advantage of replication slots.

==Logical decoding ==

TODO: Blog posts, explanation of what it's the groundwork for

==Performance improvements==

===GIN indexes now faster and smaller ===

TODO: Blog posts, benchmark results

===pg_prewarm===

When a database instance is restarted, its shared buffers are empty again, which means all queries will initially have to read data in direct from disk. pg_prewarm can load relation data back into the buffers to "warm" the buffers back up again. This will mean that queries, that would otherwise have to load parts of a table in bit by bit, can have the data available in shared buffers ready for them.

TODO: Example

=Other new features=

==ALTER SYSTEM==

Normally, cluster-wide settings need to be edited in the postgresql.conf, but now changes can be made via the ALTER SYSTEM command.

Example:

postgres=# ALTER SYSTEM SET log_min_duration_statement = '5s';

This doesn't actually change postgresql.conf. Instead it writes the setting to a file called postgresql.auto.conf. This file '''*always*''' gets read last, so it will always override any settings in postgresql.conf. Setting it to DEFAULT removes the line from postgresql.auto.conf:

postgres=# ALTER SYSTEM SET log_min_duration_statement = DEFAULT;

An advantage of making changes this way is that the settings are more likely to be correct as they are validated before they are added:

postgres=# ALTER SYSTEM SET wal_level = 'like the legend of the phoenix';
ERROR: invalid value for parameter "wal_level": "like the legend of the phoenix"
HINT: Available values: minimal, archive, hot_standby, logical.

If this had been set in postgresql.conf instead, and someone attempted to restart the cluster, it would fail. Naturally postgresql.auto.conf should never be edited manually.

==REFRESH MATERIALIZED VIEW CONCURRENTLY==

Prior to PostgreSQL 9.4, refreshing a materialized view meant locking the entire table, and therefore preventing anything querying it, and if a refresh took a long time to acquire the exclusive lock (while it waits for queries using it to finish), it in turn is holding up subsequent queries. This can now been mitigated with the CONCURRENTLY keyword:

postgres=# REFRESH MATERIALIZED VIEW CONCURRENTLY mv_data;

A unique index will need to exist on the materialized view though. Instead of locking the materialized view up, it instead creates a temporary updated version of it, compares the two versions, then applies INSERTs and DELETEs against the materialized view to apply the difference. This means queries can still use the materialized view while it's being updated.

Unlike its non-concurrent form, tuples aren't frozen, and it needs VACUUMing due to the aforementioned DELETEs that will leave dead tuples behind.

==WITH ORDINALITY==

Set-returning functions are most frequently used in the FROM clause of a query, and now if it's suffixed with WITH ORDINALITY, a column containing an ordered sequence of numbers is added.

For example:

postgres=# SELECT * FROM json_object_keys('{"mobile": 4234234232, "email": "x@me.com", "address": "1 Street Lane"}'::json) WITH ORDINALITY;
json_object_keys | ordinality
------------------+------------
mobile | 1
email | 2
address | 3
(3 rows)

==Ordered-set aggregates==

There is now support for ordered-set aggregates which provide order-sensitive information on columns. One example is the mode average:

postgres=# SELECT mode() WITHIN GROUP (ORDER BY eye_colour) FROM population;
mode
-------
brown
(1 row)

Or get the value of a column the specified percentage in, such as 20%

postgres=# SELECT percentile_disc(0.2) WITHIN GROUP (ORDER BY age) FROM population;
percentile_disc
-----------------
31
(1 row)

=Backwards compatibility=

[[Category:PostgreSQL_9.4]]
[[Category:What's_new_in_this_release]]

Index Maintenance

2014-04-02T12:23:31Z

Intgr: Change "unique" column to be per-index not per-relation. The current "unique" column tells you whether the table has any unique indexes at all, which is not very useful in this query

One day, you will probably need to cope with [http://www.postgresql.org/docs/current/static/routine-reindex.html routine reindexing] on your database, particularly if you don't use VACUUM aggressively enough. A particularly handy command in this area is [http://www.postgresql.org/docs/8.3/static/sql-cluster.html CLUSTER], which can help with other types of cleanup.

Avoid using [[VACUUM FULL]] in versions 8.4 and earlier.

== Index summary ==

Here's a sample query to pull the number of rows, indexes, and some info about those indexes for each table. (Only works on 8.3; ditch the pg_size_pretty if you’re on an earlier version)

{{SnippetInfo|Index summary|lang=SQL|version=>=8.1|category=Performance}}
<source lang="sql">
SELECT
pg_class.relname,
pg_size_pretty(pg_class.reltuples::bigint) AS rows_in_bytes,
pg_class.reltuples AS num_rows,
count(indexname) AS number_of_indexes,
CASE WHEN x.is_unique = 1 THEN 'Y'
ELSE 'N'
END AS UNIQUE,
SUM(case WHEN number_of_columns = 1 THEN 1
ELSE 0
END) AS single_column,
SUM(case WHEN number_of_columns IS NULL THEN 0
WHEN number_of_columns = 1 THEN 0
ELSE 1
END) AS multi_column
FROM pg_namespace
LEFT OUTER JOIN pg_class ON pg_namespace.oid = pg_class.relnamespace
LEFT OUTER JOIN
(SELECT indrelid,
max(CAST(indisunique AS integer)) AS is_unique
FROM pg_index
GROUP BY indrelid) x
ON pg_class.oid = x.indrelid
LEFT OUTER JOIN
( SELECT c.relname AS ctablename, ipg.relname AS indexname, x.indnatts AS number_of_columns FROM pg_index x
JOIN pg_class c ON c.oid = x.indrelid
JOIN pg_class ipg ON ipg.oid = x.indexrelid )
AS foo
ON pg_class.relname = foo.ctablename
WHERE
pg_namespace.nspname='public'
AND pg_class.relkind = 'r'
GROUP BY pg_class.relname, pg_class.reltuples, x.is_unique
ORDER BY 2;
</source>

== Index size/usage statistics ==

Table & index sizes along which indexes are being scanned and how many tuples are fetched. See [[Disk Usage]] for another view that includes both table and index sizes.

{{SnippetInfo|Index statistics|lang=SQL|version=>=8.1|category=Performance}}
<source lang="sql">
SELECT
t.tablename,
indexname,
c.reltuples AS num_rows,
pg_size_pretty(pg_relation_size(quote_ident(t.tablename)::text)) AS table_size,
pg_size_pretty(pg_relation_size(quote_ident(indexrelname)::text)) AS index_size,
CASE WHEN indisunique THEN 'Y'
ELSE 'N'
END AS unique,
idx_scan AS number_of_scans,
idx_tup_read AS tuples_read,
idx_tup_fetch AS tuples_fetched
FROM pg_tables t
LEFT OUTER JOIN pg_class c ON t.tablename=c.relname
LEFT OUTER JOIN
( SELECT c.relname as ctablename, ipg.relname as indexname, x.indnatts as number_of_columns, idx_scan, idx_tup_read, idx_tup_fetch, indexrelname, indisunique FROM pg_index x
JOIN pg_class c ON c.oid = x.indrelid
JOIN pg_class ipg ON ipg.oid = x.indexrelid
JOIN pg_stat_all_indexes psai ON x.indexrelid = psai.indexrelid )
as foo
ON t.tablename = foo.ctablename
WHERE t.schemaname='public'
order by 1,2;
</source>

== Duplicate indexes ==
Finds multiple indexes that have the same set of columns, same opclass, expression and predicate -- which make them equivalent. '''Usually''' it's safe to drop one of them, but I give no guarantees. :)

<source lang="sql">
SELECT pg_size_pretty(sum(pg_relation_size(idx))::bigint) as size,
(array_agg(idx))[1] as idx1, (array_agg(idx))[2] as idx2,
(array_agg(idx))[3] as idx3, (array_agg(idx))[4] as idx4
FROM (
SELECT indexrelid::regclass as idx, (indrelid::text ||E'\n'|| indclass::text ||E'\n'|| indkey::text ||E'\n'||
coalesce(indexprs::text,'')||E'\n' || coalesce(indpred::text,'')) as key
FROM pg_index) sub
GROUP BY key HAVING count(*)>1
ORDER BY sum(pg_relation_size(idx)) DESC;
</source>

== Index Bloat ==

One of the common needs for a REINDEX is when indexes become bloated due to either sparse deletions or use of VACUUM FULL. An estimator for the amount of bloat in a table has been included in the [http://bucardo.org/wiki/Check_postgres check_postgres] script, which you can call directly or incorporate into a larger monitoring system. Scripts based on this code and/or its concepts from other sources include:
* [http://pgsql.tapoueh.org/site/html/news/20080131.bloat.html bloat view] (Dimitri Fontaine)
* [http://www.pgcon.org/2009/schedule/events/153.en.html Visualizing Postgres] - index_byte_sizes view (Michael Glaesemann, myYearbook)
* [http://labs.omniti.com/trac/pgtreats/browser/trunk/tools OmniTI Tasty Treats for PostgreSQL] - shell and Perl pg_bloat_report scripts

== Unused Indexes ==

Since indexes add significant overhead to any table change operation, they should be removed if they are not being used for either queries or constraint enforcement (such as making sure a value is unique). How to find such indexes:

* [http://www.xzilla.net/blog/2008/Jul/Index-pruning-techniques.html Index pruning techniques]
* [http://hype-free.blogspot.com/2008/09/finding-unused-indexes-in-postgresql.html Finding unused indexes]
* [http://it.toolbox.com/blogs/database-soup/finding-useless-indexes-28796 Finding useless indexes]
* [http://jmorano.moretrix.com/2014/02/postgresql-monitor-unused-indexes/ 'Monitor unused indexes' by Johnny Morano]

== References ==

* Index statistics queries from [http://www.baconandtech.com/2009/06/06/book-review-part-i-refactoring-sql-applications-with-bonus-queries/ "Refactoring SQL Applications" review]

[[Category:Administration]][[Category:Performance]]

Performance Optimization

2014-02-12T10:15:49Z

Intgr: This article is obsolete; talks about Postgres 8.2 and manual VACUUMing. Most other articles cover shared_buffers and effective_cache_size anyway

== General Setup and Optimization ==
* [[Tuning Your PostgreSQL Server]] by Greg Smith, Robert Treat, and Christopher Browne
* [http://www.revsys.com/writings/postgresql-performance.html Performance Tuning PostgreSQL] by Frank Wiles
* [http://www.pgcon.org/2008/schedule/events/104.en.html GUCs: A Three Hour Tour] by Josh Berkus. Also useful here is his [http://pgfoundry.org/docman/view.php/1000106/84/calcfactors.sxc tuning OpenOffice spreadsheet], which suggests tuning values for 5 different types of workloads.
* [http://linuxfinances.info/info/quickstart.html QuickStart Guide to Tuning PostgreSQL] by Christopher Browne
* [http://www.varlena.com/GeneralBits/Tidbits/annotated_conf_e.html Annotated postgresql.conf] by Josh Berkus and Shridhar Daithankar (older V7.4 targeted version of material covered in the GUC tour referenced above)
* [http://www.varlena.com/GeneralBits/Tidbits/perf.html Performance Tuning] by Josh Berkus and Shridhar Daithankar
* [http://www.zope.org/Members/pupq/pg_in_aggregates Replacing Slow Loops in PostgreSQL] by Joel Burton
* [http://momjian.us/main/writings/pgsql/hw_performance/ PostgreSQL Hardware Performance Tuning] by Bruce Momjian
* [http://www.targeted.org/articles/databases/fragmentation.html The effects of data fragmentation in a mixed load database] by Dmitry Dvoinikov
* [http://www.2ndquadrant.com/static/2quad/media/pdfs/talks/Postgres_Performance_Update83.pdf PostgreSQL Performance Features in 8.3] by Simon Riggs
* [http://www.2ndquadrant.com/static/2quad/media/pdfs/talks/Postgres_Performance_Update84.pdf PostgreSQL Performance Features in 8.4] by Simon Riggs

Performance courses are available from a number of companies. Check [http://www.postgresql.org/about/eventarchive events and trainings] for further details.

==Critical maintenance for performance==
*[[Introduction to VACUUM, ANALYZE, EXPLAIN, and COUNT]] by Jim Nasby.
*[[VACUUM FULL]] and why you should avoid it
*[[Planner Statistics]]
*[[Using EXPLAIN]]
*[[Logging Difficult Queries]]
*[[Logging Checkpoints]]
*[http://www.westnet.com/~gsmith/content/postgresql/chkp-bgw-83.htm Checkpoints and the Background Writer: PostgreSQL 8.3 Improvements and Migration] by Greg Smith
*[[Bulk Loading and Restores]]
*[[Performance Analysis Tools]] by Craig Ringer

== Database architecture ==
* [[Priorities|Limiting and prioritizing user/query/database resource usage]] by Craig Ringer
* [[Prioritizing databases by separating into multiple clusters]] by Craig Ringer
* [[Clustering]]
* [[Shared Storage]]

==Database Hardware Selection and Setup==
* [[Database Hardware]]
* [[Reliable Writes]]

==Benchmark Workloads==
* [[:Category:Benchmarking]]

[[Category:Administration]][[Category:Performance]][[Category:Benchmarking]]
[[Category:General articles and guides]]

Streaming Replication

2013-08-14T12:41:23Z

Intgr: It's extremely crazy to grant superuser access with "trust" method, and isn't required in 9.1+ versions

'''Streaming Replication''' (SR) provides the capability to continuously ship and
apply the [http://www.postgresql.org/docs/current/static/wal.html WAL XLOG] records to some number of standby servers in order to keep them current.

This feature was added to PostgreSQL 9.0. The discussion below is a developer oriented one that contains some out of date information. Users of this feature should use the documentation for the feature or a setup tutorial instead:

* [http://www.postgresql.org/docs/current/static/warm-standby.html PostgreSQL Streaming Replication Documentation] - this documentation is for the latest version, but provides links you can use to look up the docs for your version.

* [[Binary Replication Tutorial]] provides an introduction to using this replication feature.

* The related but independent [[Hot Standby]] feature.

= Developer and historical details on the project =
SR was developed for inclusion in PostgreSQL 9.0 by NTT OSS Center. The lead developer is [mailto:masao.fujii@gmail.com Masao Fujii]. [http://www.pgcon.org/2008/schedule/events/76.en.html Synchronous Log Shipping Replication Presentation] introduces the early design of the feature.

= Usage =
== Users Overview ==
* '''Log-shipping'''
** XLOG records generated in the primary are periodically shipped to the standby via the network.
** In the existing warm standby, only records in a filled file are shipped, what's referred to as file-based log-shipping. In SR, XLOG records in partially-filled XLOG file are shipped too, implementing record-based log-shipping. This means the window for data loss in SR is usually smaller than in warm standby, unless the warm standby was also configured for record-based shipping (which is complicated to setup).
** The content of XLOG files written to the standby are exactly the same as those on the primary. XLOG files shipped can be used for a normal recovery and PITR.
* '''Multiple standbys'''
** More than one standby can establish a connection to the primary for SR. XLOG records are concurrently shipped to all these standbys. The delay/death of a standby does not harm log-shipping to other standbys.
** The maximum number of standbys can be specified as a GUC variable.
* '''Continuous recovery'''
** The standby continuously replays XLOG records shipped without using pg_standby.
** XLOG records shipped are replayed as soon as possible without waiting until XLOG file has been filled. The combination of [[Hot Standby]] and SR would make the latest data inserted into the primary visible in the standby almost immediately.
** The standby periodically removes old XLOG files which are no longer needed for recovery, to prevent excessive disk usage.
* '''Setup'''
** The start of log-shipping does not interfere with any query processing on the primary.
** The standby can be started in various conditions.
*** If there are XLOG files in archive directory and restore_command is supplied, at first those files are replayed. Then the standby requests XLOG records following the last applied one to the primary. This prevents XLOG files already present in the standby from being shipped again. Similarly, XLOG files in pg_xlog are also replayed before starting log-shipping.
*** If there is no XLOG files on the standby, the standby requests XLOG records following the starting XLOG location of recovery (the redo starting location).
* '''Connection settings and authentication'''
** A user can configure the same settings as a normal connection to a connection for SR (e.g., keepalive, pg_hba.conf).
* '''Activation'''
** The standby can keep waiting for activation as long as a user likes. This prevents the standby from being automatically brought up by failure of recovery or network outage.
* '''Progress report'''
** The primary and standby report the progress of log-shipping in PS display.
* '''Graceful shutdown'''
** When smart/fast shutdown is requested, the primary waits to exit until XLOG records have been sent to the standby, up to the shutdown checkpoint record.

== Restrictions ==
* '''Synchronous log-shipping'''
** By default, SR supports operates in asynchronous manner, so the commit command might return a "success" to a client before the corresponding XLOG records are shipped to the standby. To enable synchronous replication, see [http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION Synchronous Replication]
* '''Replication beyond timeline'''
** A user has to get a fresh backup whenever making the old standby catch up.
* '''Clustering'''
** Postgres doesn't provide any clustering feature.

== How to Use ==
* '''1.''' Install postgres in the primary and standby server as usual. This requires only ''configure'', ''make'' and ''make install''.
* '''2.''' Create the initial database cluster in the primary server as usual, using ''initdb''.
* '''3.''' Set up connections and authentication so that the standby server can successfully connect to the ''replication'' pseudo-database on the primary.
$ $EDITOR postgresql.conf

listen_addresses = '192.168.0.10'

$ $EDITOR pg_hba.conf

# The standby server must connect with a user that has replication privileges.
host replication replication 192.168.0.20/22 trust
* '''4.''' Set up the streaming replication related parameters on the primary server.
$ $EDITOR postgresql.conf

# To enable read-only queries on a standby server, wal_level must be set to
# "hot_standby". But you can choose "archive" if you never connect to the
# server in standby mode.
wal_level = hot_standby

# Set the maximum number of concurrent connections from the standby servers.
max_wal_senders = 5

# To prevent the primary server from removing the WAL segments required for
# the standby server before shipping them, set the minimum number of segments
# retained in the pg_xlog directory. At least wal_keep_segments should be
# larger than the number of segments generated between the beginning of
# online-backup and the startup of streaming replication. If you enable WAL
# archiving to an archive directory accessible from the standby, this may
# not be necessary.
wal_keep_segments = 32

# Enable WAL archiving on the primary to an archive directory accessible from
# the standby. If wal_keep_segments is a high enough number to retain the WAL
# segments required for the standby server, this is not necessary.
archive_mode = on
archive_command = 'cp %p /path_to/archive/%f'
* '''5.''' Start postgres on the primary server.
* '''6.''' Make a base backup by copying the primary server's data directory to the standby server.
$ psql -c "SELECT pg_start_backup('label', true)"
$ rsync -ac ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid
$ psql -c "SELECT pg_stop_backup()"
* '''7.''' Set up replication-related parameters, connections and authentication in the standby server like the primary, so that the standby might work as a primary after failover.
* '''8.''' Enable read-only queries on the standby server. But if wal_level is ''archive'' on the primary, leave hot_standby unchanged (i.e., off).
$ $EDITOR postgresql.conf

hot_standby = on
* '''9.''' Create a recovery command file in the standby server; the following parameters are required for streaming replication.
$ $EDITOR recovery.conf
# Note that recovery.conf must be in $PGDATA directory.

# Specifies whether to start the server as a standby. In streaming replication,
# this parameter must to be set to on.
standby_mode = 'on'

# Specifies a connection string which is used for the standby server to connect
# with the primary.
primary_conninfo = 'host=192.168.0.10 port=5432 user=postgres'

# Specifies a trigger file whose presence should cause streaming replication to
# end (i.e., failover).
trigger_file = '/path_to/trigger'

# Specifies a command to load archive segments from the WAL archive. If
# wal_keep_segments is a high enough number to retain the WAL segments
# required for the standby server, this may not be necessary. But
# a large workload can cause segments to be recycled before the standby
# is fully synchronized, requiring you to start again from a new base backup.
restore_command = 'cp /path_to/archive/%f "%p"'
* '''10.''' Start postgres in the standby server. It will start streaming replication.
* '''11.''' You can calculate the replication lag by comparing the current WAL write location on the primary with the last WAL location received/replayed by the standby. They can be retrieved using ''pg_current_xlog_location'' on the primary and the ''pg_last_xlog_receive_location''/''pg_last_xlog_replay_location'' on the standby, respectively.
$ psql -c "SELECT pg_current_xlog_location()" -h192.168.0.10 (primary host)
pg_current_xlog_location
--------------------------
0/2000000
(1 row)

$ psql -c "select pg_last_xlog_receive_location()" -h192.168.0.20 (standby host)
pg_last_xlog_receive_location
-------------------------------
0/2000000
(1 row)

$ psql -c "select pg_last_xlog_replay_location()" -h192.168.0.20 (standby host)
pg_last_xlog_replay_location
------------------------------
0/2000000
(1 row)
* '''12.''' You can also check the progress of streaming replication by using ''ps'' command.
# The displayed LSNs indicate the byte position that the standby server has
# written up to in the xlogs.
[primary] $ ps -ef | grep sender
postgres 6879 6831 0 10:31 ? 00:00:00 postgres: wal sender process postgres 127.0.0.1(44663) streaming 0/2000000

[standby] $ ps -ef | grep receiver
postgres 6878 6872 1 10:31 ? 00:00:01 postgres: wal receiver process streaming 0/2000000
* How to do failover
** Create the trigger file in the standby after the primary fails.
* How to stop the primary or the standby server
** Shut down it as usual (''pg_ctl stop'').
* How to restart streaming replication after failover
** Repeat the operations from '''6th'''; making a fresh backup, some configurations and starting the original primary as the standby. The primary server doesn't need to be stopped during these operations.
* How to restart streaming replication after the standby fails
** Restart postgres in the standby server after eliminating the cause of failure.
* How to disconnect the standby from the primary
** Create the trigger file in the standby while the primary is running. Then the standby would be brought up.
* How to re-synchronize the stand-alone standby after isolation
** Shut down the standby as usual. And repeat the operations from '''6th'''.
* If you have more than one slave, promoting one will break the other(s). Update their recovery.conf settings to point to the new master, set recovery_target_timeline to 'latest', scp/rsync the pg_xlog directory, and restart the slave.

= Todo =
== v9.0 ==

Moved to [[PostgreSQL_9.0_Open_Items]]

=== Committed ===
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01455.php Retrying from archive and some refactoring around Read/FetchRecord().] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00395.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg02601.php SR wrongly treats the WAL-boundary.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00396.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01715.php Adjust SR for some later changes about wal-skipping.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00399.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00024.php VACUUM FULL unexpectedly writes an XLOG UNLOGGED record.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00038.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01754.php Add a message type header.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00037.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01536.php Documentation: Add a new "Replication" chapter.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00115.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00350.php Failed assertion during recovery of partial WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00124.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00712.php A PANIC error might occur in the standby because of a partially-filled archived WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00137.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00330.php Improve the standby messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00140.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01672.php pq_getbyte_if_available() is not working because the win32 socket emulation layer simply wasn't designed to deal with non-blocking sockets.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00198.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01488.php Walsender might emit unfit messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00239.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01236.php Streaming replication on win32, still broken.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00270.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00992.php Create new section for recovery.conf.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00295.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01824.php Assertion failure in walreceiver.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00356.php commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01717.php Forbid a startup of walsender during recovery, and emit a suitable message? Or allow walsender to be started also during recovery?] - [http://archives.postgresql.org/message-id/20100316090955.9A5107541D0@cvs.postgresql.org commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01003.php How do we clean down the archive without using pg_standby?] - [http://archives.postgresql.org/message-id/20100318091718.BC14D7541D0@cvs.postgresql.org commit]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01510.php File-based log shipping without pg_standby doesn't replay the WAL files in pg_xlog.] - [http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php commit]

== v9.1 ==
=== Synchronization capability ===
* Introduce the replication mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'' and ''fsync''.
** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.
** ''recv'' or ''fsync'' makes transaction commit wait for XLOG to be received or fsynced by the standby, respectively.
** (''apply'' makes transaction commit wait for XLOG to be replayed by the standby. This mode will be supported in v9.2 or later)
** The replication mode is specified in recovery.conf of the standby as well as other parameters for replication.
*** The startup process reads the replication mode from recovery.conf and shares it to walreceiver via new shared-memory variable.
*** Walreceiver also shares it to walsender by using the replication handshake message (existing protocol needs to be extended).
** Based on the replication mode, walreceiver sends the reply meaning that replication is done up to the specified location to the primary.
*** In async, walreceiver doesn't need to send any reply other than end-of-replication message.
*** In recv or fsync, walreceiver sends the reply just after receiving or flushing XLOG, respectively.
*** New message type for the reply needs to be defined. The reply is sent as CopyData message.
** Walreceiver writes all the outstanding XLOG to disk before shutting down.
** Walsender receives the reply from the standby, updates the location of the last record replicated, and announces completion of replication.
*** New shared-memory variable to keep that location is required.
** When processing the commit command, backend waits for XLOG to be replicated to only the standbys which are in the recv or fsync replication mode.
*** Also smart shutdown waits for XLOG of shutdown checkpoint to be replicated.
* Required optimization
** Walsender should send outstanding XLOG without waiting wal_sender_delay.
*** When processing the commit command, backend signals walsender to send outstanding XLOG immediately.
** Backend should exit the wait loop as soon as the reply arrives at the primary.
*** When receiving the reply, walsender signals backends to get up from the sleep and determine whether to exit the wait loop by checking the location of the last XLOG replicated.
*** Only backends waiting for XLOG to be replicated up to the location contained in the reply are sent the signal.
** Walsender waits for the signal from backends and the reply from the standby at the same time, by using select/poll.
** Walsender reads XLOG from not only disk but also shared memory (wal buffers).
** Walreceiver should flush XLOG file only when XLOG file is switched or the related page is flushed.
*** When startup process or bgwriter flushes the buffer page, it checks whether the related XLOG has already been flushed via shared memory (location of the last XLOG flushed).
*** It flushes the buffer page, if XLOG file has already been flushed.
*** It signals walreceiver to flush XLOG file immediately and waits for the flush to complete, if XLOG file has not been flushed yet.
** While the standby is catching up with the primary, those servers should ignore the replication mode and perform asynchronous replication.
*** After those servers have almost gotten into synchronization, they perform replication based on the specified replication mode.
*** New replication states like 'catching-up', 'sync', etc need to be defined, and the state machine for them is required on both servers.
*** Current replication state can be monitored on both servers via SQL.
* Required timeout
** Add new parameter replication_timeout which is the maximum time to wait until XLOG is replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.

== Future release ==
* '''Synchronization capability'''
** Introduce the synchronization mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'', ''fsync'' and ''apply''.
*** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.
*** ''recv'', ''fsync'' and ''apply'' makes transaction commit wait for XLOG records to be received, fsynced and applied on the standby, respectively.
** Change walsender to be able to read XLOG from not only the disk but also shared memory.
** Add new parameter replication_timeout which is the maximum time to wait until XLOG records are replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.
* '''Monitoring'''
** Provide the capability to check the progress and gap of streaming replication via one query. A collaboration of HS and SR is necessary to provide that capability on the standby side.
** Provide the capability to check if the specified repliation is in progress via a query. Also more detailed status information might be necessary, e.g, the standby is catching up now, has already gotten into sync, and so on.
** Change the stats collector to collect the statistics information about replication, e.g., average delay of replication time.
** Develop the tool to calculate the latest XLOG position from XLOG files. This is necessary to check the gap of replication after the server fails.
** Also develop the tool to extract the user-readable contents from XLOG files. This is necessary to see the contents of the gap, and manually restore them.
* '''Easy to Use'''
** Introduce the parameters like:
*** replication_halt_timeout - replication will halt if no data has been sent for this much time.
*** replication_halt_segments - replication will halt if number of WAL files in pg_xlog exceeds this threshold.
*** These parameters allow us to avoid disk overflow.
** Add new feature which transfers also base backup via the direct connection between the primary and the standby.
** Add new hooks like walsender_hook and walreceiver_hook to cooperate with the add-on program for compression like pglesslog.
** Provide a graceful termination of replication via a query on the primary. On the standby, a trigger file mechanism already provides that capability.
** Support replication beyond timeline. The timeline history files need to be shipped from the primary to the standby.
* '''Robustness'''
** Support keepalive in libpq. This is useful for a client and the standby to detect a failure of the primary immediately.
* '''Miscellaneous'''
** Standalone walreceiver tool, which connects to the primary, continuously receives and writes XLOG records, independently from postgres server.
** Cascade streaming replication. Allow walsender to send XLOG to another standby during recovery.
** WAL archiving during recovery.

[[Category:Replication]]

Extension build troubleshooting

2013-05-28T19:39:25Z

Intgr: /* Troubleshooting */ replace hostname with foobar

Usually all that's necessary to build an extension is the following commands:

make
make install
make installcheck

Followed by a SQL command to create the extension in a database (replace <tt>foobar</tt> with the name of the extension):

CREATE EXTENSION foobar;

== Troubleshooting ==

If you encounter an error such as:

"Makefile", line 8: Need an operator

You need to use GNU make, which may well be installed on your system as gmake:

gmake
gmake install
gmake installcheck

If you encounter an error such as:

make: pg_config: Command not found

Be sure that you have pg_config installed and in your path. If you used a package management system such as RPM to install PostgreSQL, be sure that the -devel package is also installed. If necessary tell the build process where to find it:

env PG_CONFIG=/path/to/pg_config make && make installcheck && make install

If you encounter an error such as:

ERROR: must be owner of database regression

You need to run the test suite using a super user, such as the default "postgres" super user:

make installcheck PGUSER=postgres

Once the extension is installed, you can add it to a database. If you're running PostgreSQL 9.1.0 or greater, it's a simple as connecting to a database as a super user and running (replace <tt>foobar</tt> with the name of the extension):

CREATE EXTENSION foobar;

If you've upgraded your cluster to PostgreSQL 9.1 and already had the extension installed, you can upgrade it to a properly packaged extension with:

CREATE EXTENSION foobar FROM unpackaged;

For versions of PostgreSQL less than 9.1.0, you'll need to run the installation script (replace <tt>foobar</tt> with the name of the extension):

psql -d mydb -f /path/to/pgsql/share/contrib/foobar.sql

If you want to install the extension and all of its supporting objects into a specific schema, use the PGOPTIONS environment variable to specify the schema, like so:

PGOPTIONS=--search_path=extensions psql -d mydb -f foobar.sql

== Credits ==

This troubleshooting guide was originally written by '''David E. Wheeler'''. Copied from his [http://pgxn.org/dist/hostname/ hostname extension] README.

Extension build troubleshooting

2013-05-28T19:37:37Z

Intgr: clarify

Usually all that's necessary to build an extension is the following commands:

make
make install
make installcheck

Followed by a SQL command to create the extension in a database (replace <tt>foobar</tt> with the name of the extension):

CREATE EXTENSION foobar;

== Troubleshooting ==

If you encounter an error such as:

"Makefile", line 8: Need an operator

You need to use GNU make, which may well be installed on your system as gmake:

gmake
gmake install
gmake installcheck

If you encounter an error such as:

make: pg_config: Command not found

Be sure that you have pg_config installed and in your path. If you used a package management system such as RPM to install PostgreSQL, be sure that the -devel package is also installed. If necessary tell the build process where to find it:

env PG_CONFIG=/path/to/pg_config make && make installcheck && make install

If you encounter an error such as:

ERROR: must be owner of database regression

You need to run the test suite using a super user, such as the default "postgres" super user:

make installcheck PGUSER=postgres

Once the extension is installed, you can add it to a database. If you're running PostgreSQL 9.1.0 or greater, it's a simple as connecting to a database as a super user and running (replace <tt>foobar</tt> with the name of the extension):

CREATE EXTENSION foobar;

If you've upgraded your cluster to PostgreSQL 9.1 and already had hostname installed, you can upgrade it to a properly packaged extension with:

CREATE EXTENSION hostname FROM unpackaged;

For versions of PostgreSQL less than 9.1.0, you'll need to run the installation script (replace <tt>foobar</tt> with the name of the extension):

psql -d mydb -f /path/to/pgsql/share/contrib/foobar.sql

If you want to install the extension and all of its supporting objects into a specific schema, use the PGOPTIONS environment variable to specify the schema, like so:

PGOPTIONS=--search_path=extensions psql -d mydb -f foobar.sql

== Credits ==

This troubleshooting guide was originally written by '''David E. Wheeler'''. Copied from his [http://pgxn.org/dist/hostname/ hostname extension] README.

Extension build troubleshooting

2013-05-28T19:33:13Z

Intgr: Created page with "Usually all that's necessary to build an extension is the following commands: make make install make installcheck Followed by a SQL DDL command to create the extension in a …"

Usually all that's necessary to build an extension is the following commands:

make
make install
make installcheck

Followed by a SQL DDL command to create the extension in a database:

CREATE EXTENSION foobar;

== Troubleshooting ==

If you encounter an error such as:

"Makefile", line 8: Need an operator

You need to use GNU make, which may well be installed on your system as gmake:

gmake
gmake install
gmake installcheck

If you encounter an error such as:

make: pg_config: Command not found

Be sure that you have pg_config installed and in your path. If you used a package management system such as RPM to install PostgreSQL, be sure that the -devel package is also installed. If necessary tell the build process where to find it:

env PG_CONFIG=/path/to/pg_config make && make installcheck && make install

If you encounter an error such as:

ERROR: must be owner of database regression

You need to run the test suite using a super user, such as the default "postgres" super user:

make installcheck PGUSER=postgres

Once the extension is installed, you can add it to a database. If you're running PostgreSQL 9.1.0 or greater, it's a simple as connecting to a database as a super user and running (replace <tt>foobar</tt> with name of the extension):

CREATE EXTENSION foobar;

If you've upgraded your cluster to PostgreSQL 9.1 and already had hostname installed, you can upgrade it to a properly packaged extension with:

CREATE EXTENSION hostname FROM unpackaged;

For versions of PostgreSQL less than 9.1.0, you'll need to run the installation script (replace <tt>foobar</tt> with name of the extension):

psql -d mydb -f /path/to/pgsql/share/contrib/foobar.sql

If you want to install the extension and all of its supporting objects into a specific schema, use the PGOPTIONS environment variable to specify the schema, like so:

PGOPTIONS=--search_path=extensions psql -d mydb -f foobar.sql

== Credits ==

This troubleshooting guide was originally written by '''David E. Wheeler'''. Copied from his [http://pgxn.org/dist/hostname/ hostname extension] README.

Query column with range types

2013-04-04T20:00:22Z

Intgr: /* Code */ more blank lines

== Rationale ==

The main use case for range types is to store ranges in PostgreSQL tables, and then find rows whose range includes a certain literal. These can already be indexed using GIN and GiST index types.


db=# EXPLAIN ANALYZE SELECT * FROM rng WHERE rng @> 10.0;
Bitmap Heap Scan on rng (cost=48.40..2709.45 rows=1000 width=32) (actual time=0.103..0.103 rows=10 loops=1)
Recheck Cond: (rng @> 10.0)
-> Bitmap Index Scan on rng_rng_idx (cost=0.00..48.15 rows=1000 width=0) (actual time=0.096..0.096 rows=10 loops=1)
Index Cond: (rng @> 10.0)
Total runtime: 0.136 ms

However, you cannot index inverse queries - if you want to find single values in a certain range:


db=# EXPLAIN ANALYZE SELECT * FROM num WHERE i <@ '[0, 10)';
Seq Scan on num (cost=0.00..16925.00 rows=1000 width=6) (actual time=0.014..188.761 rows=9 loops=1)
Filter: (i <@ '[0,10]'::numrange)
Rows Removed by Filter: 999991
Total runtime: 188.820 ms

== Example ==

Of course the above query is equivalent to the indexable query <tt>WHERE num >= 0 AND num < 10</tt>. But what if you want to use range types? Well, you can create a custom operator with a function that automatically gets inlined:

db=# EXPLAIN ANALYZE SELECT * FROM num WHERE i <@ '[0, 10)';
Index Only Scan using num_i_idx on num (cost=0.00..4.56 rows=10 width=6) (actual time=0.014..0.016 rows=9 loops=1)
Index Cond: ((i >= 0::numeric) AND (i < 10::numeric))
Heap Fetches: 0
Total runtime: 0.039 ms

Infinite terms get optimized out:

db=# EXPLAIN ANALYZE SELECT * FROM num WHERE i <@ '[,10]';
Index Only Scan using num_i_idx on num (cost=0.00..4.53 rows=10 width=6) (actual time=0.007..0.009 rows=10 loops=1)
Index Cond: (i <= 10::numeric)
Heap Fetches: 0
Total runtime: 0.032 ms

=== Beware! ===

These custom operators are deliberately defined to be non-commative. If you create these operators, do not write range queries in the form <tt>10.0 <@ range_col</tt> because it will blow up:

db=# EXPLAIN ANALYZE SELECT * FROM rng WHERE 10.0 <@ rng;
Seq Scan on rng (cost=0.00..46358.16 rows=444424 width=32) (actual time=0.013..240.054 rows=10 loops=1)
Filter: ((lower_inf(rng) OR CASE WHEN lower_inc(rng) THEN (10.0 >= lower(rng)) ELSE (10.0 > lower(rng)) END)
AND (upper_inf(rng) OR CASE WHEN upper_inc(rng) THEN (10.0 <= upper(rng)) ELSE (10.0 < upper(rng)) END))
Rows Removed by Filter: 999990
Total runtime: 240.125 ms

== Code ==

You can adapt this code to any range type you need, just by changing the types of the function and the operator.

<source lang="sql">
-- int4range (integer)
CREATE OR REPLACE FUNCTION range_contains(el integer, rng int4range) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=integer, rightarg=int4range);

-- int8range (bigint)
CREATE OR REPLACE FUNCTION range_contains(el bigint, rng int8range) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=bigint, rightarg=int8range);

-- numrange (numeric)
CREATE OR REPLACE FUNCTION range_contains(el numeric, rng numrange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=numeric, rightarg=numrange);

-- tsrange (timestamp without time zone)
CREATE OR REPLACE FUNCTION range_contains(el timestamp, rng tsrange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=timestamp, rightarg=tsrange);

-- tstzrange (timestamp with time zone)
CREATE OR REPLACE FUNCTION range_contains(el timestamptz, rng tstzrange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=timestamptz, rightarg=tstzrange);

-- daterange (date)
CREATE OR REPLACE FUNCTION range_contains(el date, rng daterange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=date, rightarg=daterange);
</source>

Idea from Andres Freund, implemented by Marti Raudsepp.

Query column with range types

2013-04-04T19:59:28Z

Intgr: credits

Query column with range types

2013-04-03T21:34:51Z

Intgr: /* Beware! */

Query column with range types

2013-04-03T21:26:47Z

Intgr: Created page with "== Rationale == The main use case for range types is to store ranges in PostgreSQL tables, and then find rows whose range includes a certain literal. These can already be indexe…"

== Rationale ==

The main use case for range types is to store ranges in PostgreSQL tables, and then find rows whose range includes a certain literal. These can already be indexed using GIN and GiST index types.


db=# EXPLAIN ANALYZE SELECT * FROM rng WHERE rng @> 10.0;
Bitmap Heap Scan on rng (cost=48.40..2709.45 rows=1000 width=32) (actual time=0.103..0.103 rows=10 loops=1)
Recheck Cond: (rng @> 10.0)
-> Bitmap Index Scan on rng_rng_idx (cost=0.00..48.15 rows=1000 width=0) (actual time=0.096..0.096 rows=10 loops=1)
Index Cond: (rng @> 10.0)
Total runtime: 0.136 ms

However, you cannot index inverse queries - if you want to find single values in a certain range:


db=# EXPLAIN ANALYZE SELECT * FROM num WHERE i <@ '[0, 10)';
Seq Scan on num (cost=0.00..16925.00 rows=1000 width=6) (actual time=0.014..188.761 rows=9 loops=1)
Filter: (i <@ '[0,10]'::numrange)
Rows Removed by Filter: 999991
Total runtime: 188.820 ms

== Example ==

Of course the above query is equivalent to the indexable query <tt>WHERE num >= 0 AND num < 10</tt>. But what if you want to use range types? Well, you can create a custom operator with a function that automatically gets inlined:

db=# EXPLAIN ANALYZE SELECT * FROM num WHERE i <@ '[0, 10)';
Index Only Scan using num_i_idx on num (cost=0.00..4.56 rows=10 width=6) (actual time=0.014..0.016 rows=9 loops=1)
Index Cond: ((i >= 0::numeric) AND (i < 10::numeric))
Heap Fetches: 0
Total runtime: 0.039 ms

Infinite terms get optimized out:

db=# EXPLAIN ANALYZE SELECT * FROM num WHERE i <@ '[,10]';
Index Only Scan using num_i_idx on num (cost=0.00..4.53 rows=10 width=6) (actual time=0.007..0.009 rows=10 loops=1)
Index Cond: (i <= 10::numeric)
Heap Fetches: 0
Total runtime: 0.032 ms

=== Beware! ===

These custom operators are deliberately defined to be non-commative. Do not write range queries in the form <tt>10.0 <@ range_col</tt> because it will blow up:

db=# EXPLAIN ANALYZE SELECT * FROM rng WHERE 10.0 <@ rng;
Seq Scan on rng (cost=0.00..46358.16 rows=444424 width=32) (actual time=0.013..240.054 rows=10 loops=1)
Filter: ((lower_inf(rng) OR CASE WHEN lower_inc(rng) THEN (10.0 >= lower(rng)) ELSE (10.0 > lower(rng)) END) AND (upper_inf(rng) OR CASE WHEN upper_inc(rng) THEN (10.0 <= upper(rng)) ELSE (10.0 < upper(rng)) END))
Rows Removed by Filter: 999990
Total runtime: 240.125 ms

== Code ==

You can adapt this code to any range type you need, just by changing the types of the function and the operator.

<source lang="sql">
-- int4range (integer)
CREATE OR REPLACE FUNCTION range_contains(el integer, rng int4range) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=integer, rightarg=int4range);

-- int8range (bigint)
CREATE OR REPLACE FUNCTION range_contains(el bigint, rng int8range) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=bigint, rightarg=int8range);

-- numrange (numeric)
CREATE OR REPLACE FUNCTION range_contains(el numeric, rng numrange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=numeric, rightarg=numrange);

-- tsrange (timestamp without time zone)
CREATE OR REPLACE FUNCTION range_contains(el timestamp, rng tsrange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=timestamp, rightarg=tsrange);

-- tstzrange (timestamp with time zone)
CREATE OR REPLACE FUNCTION range_contains(el timestamptz, rng tstzrange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=timestamptz, rightarg=tstzrange);

-- daterange (date)
CREATE OR REPLACE FUNCTION range_contains(el date, rng daterange) RETURNS bool IMMUTABLE LANGUAGE sql AS $$
SELECT (lower_inf(rng) OR (CASE WHEN lower_inc(rng) THEN el >= lower(rng) ELSE el > lower(rng) END))
AND (upper_inf(rng) OR (CASE WHEN upper_inc(rng) THEN el <= upper(rng) ELSE el < upper(rng) END))
$$;
CREATE OPERATOR <@ (procedure=range_contains, leftarg=date, rightarg=daterange);
</source>

Using pg upgrade on Ubuntu/Debian

2012-12-20T18:07:27Z

Intgr: Remove -F, since 9.2.2 already uses synchronous_commit=off, which is safer and not much slower

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version (8.3 and up -- [http://www.postgresql.org/docs/current/static/pgupgrade.html subject to limitations in pg_upgrade]).

Simply replace any version numbers on the example command lines given below.

== WARNING! ==
'''These instructions are experimental!''' This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. Always test in a staging environment before running on production. I have tested this on Ubuntu 12.04.

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o ' -c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O ' -c config_file=/etc/postgresql/9.2/main/postgresql.conf'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

First, stop all clusters:

sudo /etc/init.d/postgresql stop

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories, where pg_upgrade expects to find them.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. If you want to get rid of it -- '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-12-10T21:49:49Z

Intgr: Work around bug in pg_upgrade 9.2.2

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version (8.3 and up -- [http://www.postgresql.org/docs/current/static/pgupgrade.html subject to limitations in pg_upgrade]).

Simply replace any version numbers on the example command lines given below.

== WARNING! ==
'''These instructions are experimental!''' This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. Always test in a staging environment before running on production. I have tested this on Ubuntu 12.04.

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o ' -c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O ' -c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

First, stop all clusters:

sudo /etc/init.d/postgresql stop

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories, where pg_upgrade expects to find them.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. If you want to get rid of it -- '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Index-only scans

2012-11-16T13:39:01Z

Intgr: Be clearer that this is PostgreSQL-specific. AFAIK some other databases do store visibility info in their indexes

Index-only scans are a major performance feature added to Postgres 9.2. They allow certain types of queries to be satisfied just by retrieving data from indexes, and not from tables. This can result in a significant reduction in the amount of I/O necessary to satisfy queries.

During a regular index scan, indexes are traversed, like any other tree structure, by comparing a constant against Datums that are stored in the index. Btree-indexed types must satisfy the trichotomy property; that is, the type must follow the reflexive, symmetric and transitive law. Those laws accord with our intuitive understanding of how a type ought to behave anyway, but the fact that an index's physical structure reflects the relative values of Datums actually mandates that these rules be followed by types. Indexes contain what are technically redundant copies of the column data that is indexed.

PostgreSQL indexes do not contain visibility information. That is, it is not possible to ascertain if any given tuple is visible to the current transaction, which is why it has taken so long for index-only scans to be implemented. Writing an implementation with a cheap but reliable visibility look-aside proved challenging.

The implementation of the feature disproportionately involved making an existing on-disk structure called the visibility map crash-safe. It was necessary for the structure to reliably (and inexpensively) indicate visibility of index tuples - to do any less would imply the possibility of index-only scans producing incorrect results, which of course would be absolutely unacceptable.

The fact that indexes only contain data that is actually indexed, and not other unindexed columns, naturally precludes using an index-only scan when the other columns are queried (by appearing in a query select list, for example).

=== Example queries where index-only scans could be used in principle ===

Assuming that there is some (non-expression) index on a column (typically a primary key):

select count(*) from categories;

Assuming that there is a composite index on (1st_indexed_col, 2nd_indexed_col):

select 1st_indexed_col, 2nd_indexed_col from categories;

Postgres 9.2 added the capability of allowing indexed_col op ANY(ARRAY[...]) conditions to be used in plain index scans and index-only scans. Previously, such conditions could only be used in bitmap index scans. For this reason, it is possible to see an index-only scan for these ScalarArrayOpExpr queries:

select indexed_col from categories where indexed_col in (4, 5, 6);

=== Index-only scans and index-access methods ===

Index-only scans are not actually limited to scans on btree indexes. SP-GiST operator classes may optionally support index-only scans (naturally, this is only sensible when the index isn't lossy).

postgres=# select amname, amcanreturn from pg_am where amcanreturn != 0;
amname | amcanreturn
--------+--------------
btree | btcanreturn
spgist | spgcanreturn
(2 rows)

Support for additional index AMs may follow in a future release of PostgreSQL.

=== The Visibility Map (and other relation forks) ===

The Visibility Map is a simple data structure associated with every heap relation (table). It is a "relation fork"; an on-disk ancillary file associated with a particular relation (table or index). Note that index relations (that is, indexes) do not have a visibility map associated with them. The visibility map is concerned with tracking which tuples are visible to all transactions at a high level. Tuples from one transaction may or may not be visible to any given other transaction, depending on whether or not their originating transaction actually committed (yet, or ever, if the transaction aborted), and when that occurred relative to our transaction's current snapshot. Note that the exact
behaviour depends on our transaction isolation level. Note also that it is quite possible for one transaction to see one physical tuple/set of values for one logical tuple, while another transaction sees other, distinct values for that same logical tuple, because, in effect, each of the two transaction has a differing idea of what constitutes "now". This is the core idea of MVCC. When there is absolute consensus that all physical tuples in a heap page are visible, the page's corresponding bit may be set.

Another relation fork that you may be familiar with is the freespace map. In contrast to the visibility map, there is a FSM for both heap and index relations (with the sole exception of hash index relations, which have none).

The purpose of the freespace map is to quickly locate a page with enough free space to hold a tuple to be stored, or to determine if no such page exists and the relation has to be extended.

In PostgreSQL 8.4, the current freespace map implementation was added. It made the freespace map an on-disk relation fork. The previous implementation required administrators to guestimate the number of relations, and the required freespace map size for each, so that the freespace map existed only in a fixed allocation of shared memory. This tended to result in wasted space due to undersizing, as the core system's storage manager needlessly extended relations.

[peter@peterlaptop 12935]$ ls -l -h -a
-rw-------. 1 peter peter 8.0K Sep 28 00:00 12910
-rw-------. 1 peter peter 24K Sep 28 00:00 12910_fsm
-rw-------. 1 peter peter 8.0K Sep 28 00:00 12910_vm
***SNIP***

The FSM is structured as a binary tree [http://www.postgresql.org/docs/9.2/static/storage-fsm.html]. There is one leaf node per heap page, with non-leaf nodes stores the maximum amount of free space for any of its children. So, unlike EXPLAIN output's node costs, the values are not cumulative.

The visibility map is a simpler structure. There is one bit for each page in the heap relation that the visibility map corresponds to.

The primary practical reason for having and maintaining the visibility map is to optimise VACUUM. A set bit indicates that all tuples on the corresponding heap page are known to be visible to all transactions, and therefore that vacuuming the page is unnecessary. Like the new freespace map implementation, the visibility map was added in Postgres 8.4.

The visibility map is conservative in that a set bit (1) indicates that all tuples are visible on the page, but an unset bit (0) indicates that that condition may or may not be true [http://www.postgresql.org/docs/9.2/static/storage-vm.html].

=== Crash safety, recovery and the visibility map ===

This involves WAL-logging setting a bit within the visibility map during VACUUM, and taking various special measures during recovery.

The Postgres write-ahead log is widely used to ensure crash-safety, but it is also intergral to the built-in Hot Standby/Streaming replication feature.

Recovery treats marking a page all-visible as a recovery conflict for snapshots that could still fail to see XIDs on that page. PostgreSQL may in the future try to soften this, so that the implementation simply forces index scans to do heap fetches in cases where this may be an issue, rather than throwing a hard conflict.

=== Covering indexes ===

Covering indexes are indexes creating for the express purpose of being used in index-only scans. They typically "cover" more columns than would otherwise make sense for an index, typically columns that are known to be part of particular expensive, frequently executed query's selectlist. PostgreSQL supports using just the first few columns of the index in a regular index scan if that is in the query's predicate, so covering indexes need not be completely useless for regular index scans.

=== Interaction with HOT ===

HOT (Heap-only tuples) is a major performance feature that was added in Postgres 8.3. This allowed UPDATES to rows (which, owing to Postgres's MVCC architecture, are implemented with a deletion and insertion of physical tuples) to only have to create a new physical heap tuple when inserting, and not a new index tuple, if and only if the update did not affect indexed columns.

With HOT, it became possible for an index scan to traverse a so-called HOT chain; it could get from the physical index tuple (which would probably have been created by an original INSERT, and related to an earlier version of the logical tuple), to the corresponding physical heap tuple. The heap tuple would itself contain a pointer to the next version of the tuple (that is, the tuple ctid), which might, in turn, have a pointer of its own. The index scan eventually arrives at tuple that is current according to the query's snapshot.

HOT also enables opportunistic mini-vacuums, where the HOT chain is "pruned".

All told, this performance optimisation has been found to be very valuable, particularly for OLTP workloads. It is quite natural that tuples that are frequently updated are generally not indexed. However, when considering creating a covering index, the need to maximise the number of HOT updates should be carefully weighed.

You can monitor the total proportion of HOT updates for each relation using this query.

postgres=# select n_tup_upd, n_tup_hot_upd from pg_stat_user_tables;

=== What types of queries may be satisfied by an index-only scan? ===

Aside from the obvious restriction that queries cannot reference columns that are not indexed by a single index in order to use an index-only scan, the need to visit the heap where all tuples are not known to be visible is relatively expensive. The planner weighs this factor heavily when considering an index-only scan, and in general the need to ensure that the bulk of the table's tuples have their visibility map bits set is likely to restrict index-only scans' usefulness to queries against infrequently updated tables.

It is not necessary for all bits to be set; index-only scans may "visit the heap" if that is necessary. Index-only scans are something of a misnomer, in fact - index mostly scans might be a more appropriate appellation. An explain analyze involving an index-only scan will indicate how frequently that occurred in practice.

postgres=# explain analyze select count(*) from categories;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=12.53..12.54 rows=1 width=0) (actual time=0.046..0.046 rows=1 loops=1)
-> Index Only Scan using categories_pkey on categories (cost=0.00..12.49 rows=16 width=0) (actual time=0.018..0.038 rows=16 loops=1)
Heap Fetches: 16
Total runtime: 0.108 ms
(4 rows)

As the number of heap "visits" goes up, the planner will eventually conclude that an index-only scan isn't desirable, as it isn't the cheapest possible plan according to its cost model. The value of index-only scans lies wholly in their potential to allow us to elide heap access (if only partially) and minimise I/O.

=== Is "count(*)" much faster now? ===

A traditional complaint made of PostgreSQL, generally when comparing it unfavourably with MySQL (at least when using the MyIsam storage engine, which doesn't use MVCC) has been "count(*) is slow". Index-only scans *can* be used to satisfy these queries without there being any predicate to limit the number of rows returned, and without forcing an index to be used by specifying that the tuples should be ordered by an indexed column. However, in practice that isn't particularly likely.

It is important to realise that the planner is concerned with minimising the total cost of the query. With databases, the cost of I/O typically dominates. For that reason, "count(*) without any predicate" queries will only use an index-only scan if the index is significantly smaller than its table. This typically only happens when the table's row width is much wider than some indexes'.

=== Why isn't my query using an index-only scan? ===

VACUUM does not have any particular tendency to behave more aggressively to facilitate using index-only scans more frequently. While VACUUM can be set to behave more aggressively in various ways, it's far from clear that to do so specifically to make index-only scans occur more frequently represents a sensible course of action.

The planner doesn't directly examine the entire visibility map of a relation when considering an index-only scan (however, the executor does maintain a running tally, which is visible in explain analyze output). However, the planner does naturally weigh the proportion of pages which are known visible to all.

In Postgres 9.2, statistics are gathered about the proportion of pages that are known all-visible. The pg_class.relallvisible column indicates how many pages are visible (the proportion can be obtained by calculating it as a proportion of pg_class.relpages). These statistics are updated during VACUUM and ANALYZE.

Note that it is possible to examine the number of index scans (including index-only scans and bitmap index scans) by examining
pg_stat_user_indexes.idx_scan. If your covering index isn't being used, you're essentially paying for the overhead of maintaining it during writes with no benefit in return. Drop the index!

=== Summary ===

It is possible for index-only scans to greatly decrease the amount of I/O required to execute some queries. For certain queries, particularly queries that are characteristic of data warehousing (i.e. relatively large amounts of static, infrequently-updated data where reports on historic data is frequently required), they can considerably improve performance. Such queries have been observed to execute anything from twice to twenty times as fast with index-only scans. However, one should bear in mind that:

* Index-only scans are opportunistic, in that they take advantage of a pre-existing state of affairs where it happens to be possible to elide heap access. However, the server doesn't make any particular effort to facilitate index-only scans, and it is difficult to recommend a course of action to make index-only scans occur more frequently, except to define covering indexes in response to a measured need (For example, when pg_stat_statements indicates that a disproportionate amount of I/O is being used to execute a query against fairly static data, with a smallish subset of table columns retrieved).

* When creating a covering index, the likely effect on HOT updates should be weighed heavily. Are there many HOT updates on the table to begin with? This is a general point of concern, because creating an index may prevent HOT updates from occurring, and because the number of HOT updates is a reasonably good proxy for just how static a table is, and therefore how likely it is that most heap pages are known to be all-visible.

* Index-only scans are only used when the planner surmises that that will reduce the total amount of I/O required, according to its imperfect cost-based modelling. This all heavily depends on visibility of tuples, if an index would be used anyway (i.e. how selective a predicate is, etc), and if there is actually an index available that could be used by an index-only scan in principle.

[[Category:Indexes]]

Using pg upgrade on Ubuntu/Debian

2012-10-29T11:19:16Z

Intgr: Supported in 8.3 and up, subject to documented limitations

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version (8.3 and up -- [http://www.postgresql.org/docs/current/static/pgupgrade.html subject to limitations in pg_upgrade]).

Simply replace any version numbers on the example command lines given below.

== WARNING! ==
'''These instructions are experimental!''' This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. Always test in a staging environment before running on production. I have tested this on Ubuntu 12.04.

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

First, stop all clusters:

sudo /etc/init.d/postgresql stop

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories, where pg_upgrade expects to find them.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. If you want to get rid of it -- '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T11:07:25Z

Intgr: /* New cluster database "XXX" is not empty */ clarify

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!''' This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. Always test in a staging environment before running on production. I have tested this on Ubuntu 12.04.

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

First, stop all clusters:

sudo /etc/init.d/postgresql stop

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories, where pg_upgrade expects to find them.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. If you want to get rid of it -- '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T11:06:40Z

Intgr: /* Upgrading to version 9.1 */ explain

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!''' This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. Always test in a staging environment before running on production. I have tested this on Ubuntu 12.04.

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

First, stop all clusters:

sudo /etc/init.d/postgresql stop

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories, where pg_upgrade expects to find them.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T11:05:44Z

Intgr: /* Upgrading to version 9.1 */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!''' This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. Always test in a staging environment before running on production. I have tested this on Ubuntu 12.04.

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

First, stop all clusters:

sudo /etc/init.d/postgresql stop

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T11:04:18Z

Intgr: fix 9.2 "prerelease" warning

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!''' This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. Always test in a staging environment before running on production. I have tested this on Ubuntu 12.04.

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T11:02:09Z

Intgr: /* Prerequisites */ remove port number coloring from 9.2 example

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T11:00:18Z

Intgr: /* Upgrading to version 9.1 */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

Note that I'm now using version numbers 9.0 and 9.1:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.0/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log

First you have to symlink the configuration files to respective PostgreSQL data directories.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T10:59:20Z

Intgr:

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Upgrading to version 9.1 ==
The above instructions don't work when you upgrade to PostgreSQL 9.1. You have to jump through some hoops.

Note that I'm now using version numbers 9.0 and span style="color:red">9.1

First you have to symlink the configuration files to respective PostgreSQL data directories.

ln -s /etc/postgresql/9.1/main/postgresql.conf /var/lib/postgresql/9.1/main/
ln -s /etc/postgresql/9.0/main/postgresql.conf /var/lib/postgresql/9.0/main/

And now you have to run the pg_upgrade script with your respective configured port numbers:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.0 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.0-main.log
9.1 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.1-main.log

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.1/bin/pg_upgrade \
-b /usr/lib/postgresql/9.0/bin \
-B /usr/lib/postgresql/9.1/bin \
-d /var/lib/postgresql/9.0/main \
-D /var/lib/postgresql/9.1/main \
-p 5432 \
-P 5433

Afterwards, you may delete the unnecessary config file links (but it's safe to leave them there if you want)

rm /var/lib/postgresql/9.1/main/postgresql.conf
rm /var/lib/postgresql/9.0/main/postgresql.conf

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-29T10:47:26Z

Intgr: /* Upgrading */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading to version 9.2 or later ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-26T12:31:15Z

Intgr: /* Where do I get PostgreSQL 9.2? */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped when a new major version is released. If you want to use this PPA, be prepared to always upgrade to the newest major version soon after it's released.

Using pg upgrade on Ubuntu/Debian

2012-10-26T09:30:21Z

Intgr: /* Where do I get PostgreSQL 9.2? */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

'''Warning:''' Support for the current major version in the PPA may be dropped earlier when a new major version is released. Be prepared to upgrade to the next major version when the older is no longer supported.

Repmgr cleanup trigger

2012-10-09T13:25:39Z

Intgr: clarify

{{SnippetInfo|repmgr cleanup trigger|lang=PL/pgSQL|version=9.0+|category=Administrative|depends=repmgr}}
The '''repmgr''' replication system requires you to run daemons on each slave to populate the repl_monitor table with mostly useless state data. In addition, it requires you to run a separate "cluster cleanup" cron job on the master to delete this useless data (hey, the same daemon couldn't possibly do that, could it?)

To simplify deployment/maintenance, that cron job can be replaced with the following trigger. (Only tested on repmgr 1.2.0)

== Cleanup trigger ==

<source lang="sql">
SET search_path=repmgr_CLUSTERNAME; -- change CLUSTERNAME to your repmgr cluster name

CREATE OR REPLACE FUNCTION repl_cleanup_proc() RETURNS trigger LANGUAGE plpgsql
SET search_path=repmgr_CLUSTERNAME
AS $$
DECLARE
cleanup_age interval = '1 day'; -- how many records to keep
cleanup_batch interval = '1 hour'; -- how many records to delete in one shot
BEGIN
IF EXISTS(SELECT * FROM repl_monitor WHERE last_monitor_time < (now() - cleanup_age - cleanup_batch)) THEN
DELETE FROM repl_monitor WHERE last_monitor_time < (now() - cleanup_age);
END IF;
RETURN new;
END;
$$;

CREATE TRIGGER repl_cleanup_trigger AFTER INSERT ON repl_monitor FOR EACH STATEMENT EXECUTE PROCEDURE repl_cleanup_proc();

-- Test the trigger (does not insert any data, but triggers deletion if necessary)
INSERT INTO repl_monitor (primary_node) SELECT null WHERE false;
</source>

[[Category:Replication]]

Repmgr cleanup trigger

2012-10-02T09:28:36Z

Intgr: +test

{{SnippetInfo|repmgr cleanup trigger|lang=PL/pgSQL|version=9.0+|category=Administrative|depends=repmgr}}
The '''repmgr''' replication system requires you to run daemons on each slave to populate the repl_monitor table with mostly useless state data. In addition, it requires you to run a separate "cluster cleanup" cron job on the master to delete this useless data (hey, the same daemon couldn't possibly do that, could it?)

To simplify deployment/maintenance, that cron job can be replaced with the following trigger. (Only tested on repmgr 1.2.0)

== Cleanup trigger ==

<source lang="sql">
SET search_path=repmgr_CLUSTERNAME; -- change CLUSTERNAME to your repmgr cluster name

CREATE OR REPLACE FUNCTION repl_cleanup_proc() RETURNS trigger LANGUAGE plpgsql
SET search_path=repmgr_CLUSTERNAME
AS $$
DECLARE
cleanup_age interval = '1 day'; -- how many records to keep
cleanup_batch interval = '1 hour'; -- how many records to delete in one shot
BEGIN
IF EXISTS(SELECT * FROM repl_monitor WHERE last_monitor_time < (now() - cleanup_age - cleanup_batch)) THEN
DELETE FROM repl_monitor WHERE last_monitor_time < (now() - cleanup_age);
END IF;
RETURN new;
END;
$$;

CREATE TRIGGER repl_cleanup_trigger AFTER INSERT ON repl_monitor FOR EACH STATEMENT EXECUTE PROCEDURE repl_cleanup_proc();

-- Test the trigger (does not insert any data)
INSERT INTO repl_monitor (primary_node) SELECT null WHERE false;
</source>

[[Category:Replication]]

Repmgr

2012-10-02T09:04:43Z

Intgr: Created page with "'''repmgr''' is a set of open source tools that helps DBAs and System administrators manage a cluster of PostgreSQL databases. See: [http://www.repmgr.org/ www.repmgr.org] [[Ca…"

'''repmgr''' is a set of open source tools that helps DBAs and System administrators manage a cluster of PostgreSQL databases.

See: [http://www.repmgr.org/ www.repmgr.org]

[[Category:Replication]]

Repmgr cleanup trigger

2012-10-02T08:14:25Z

Intgr: Created page with "{{SnippetInfo|repmgr cleanup trigger|lang=PL/pgSQL|version=9.0+|category=Administrative}} The '''repmgr''' replication system requires you to run daemons on each slave to populat…"

{{SnippetInfo|repmgr cleanup trigger|lang=PL/pgSQL|version=9.0+|category=Administrative}}
The '''repmgr''' replication system requires you to run daemons on each slave to populate the repl_monitor table with mostly useless state data. In addition, it requires you to run a "cluster cleanup" cron job on the master to delete this useless data (hey, the same daemon couldn't possibly do that, could it?)

To simplify deployment/maintenance, that cron job can be replaced with the following trigger.

== Cleanup trigger ==

<source lang="sql">
SET search_path=repmgr_CLUSTERNAME; -- change CLUSTERNAME to your repmgr cluster name

CREATE OR REPLACE FUNCTION repl_cleanup_proc() RETURNS trigger LANGUAGE plpgsql
SET search_path=repmgr_CLUSTERNAME
AS $$
DECLARE
cleanup_age interval = '1 day'; -- how many records to keep
cleanup_batch interval = '1 hour'; -- how many records to delete in one shot
BEGIN
IF EXISTS(SELECT * FROM repl_monitor WHERE last_monitor_time < (now() - cleanup_age - cleanup_batch)) THEN
DELETE FROM repl_monitor WHERE last_monitor_time < (now() - cleanup_age);
END IF;
RETURN new;
END;
$$;

CREATE TRIGGER repl_cleanup_trigger AFTER INSERT ON repl_monitor FOR EACH STATEMENT EXECUTE PROCEDURE repl_cleanup_proc();
</source>

What's new in PostgreSQL 9.2

2012-09-25T15:14:33Z

Intgr: /* Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight */ missing space

{{Languages}}

This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release – PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.

=Major new features=

==Index-only scans ==

In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.

It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.

There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.

The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).

If the data has been read only since the last VACUUM then the data is All Visible and the index only scan feature can improve performance.

Here is an example.

CREATE TABLE demo_ios (col1 float, col2 float, col3 text);

In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains won't be that big in real life.

INSERT INTO demo_ios SELECT generate_series(1,100000000),random(), 'mynotsolongstring';

SELECT pg_size_pretty(pg_total_relation_size('demo_ios'));
pg_size_pretty
----------------
6512 MB

Let's pretend that the query is this:

SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.01 AND 0.02

In order to use an index only scan on this query, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).

CREATE index idx_demo_ios on demo_ios(col2,col1);

We vacuum the table, so that the visibility map to be up-to-date:

VACUUM demo_ios;

All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).

Let's first try without Index Only Scans:

SET enable_indexonlyscan to off;

EXPLAIN (analyze,buffers) select col1,col2 FROM demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Rows Removed by Index Recheck: 68098621
Buffers: shared hit=2 read=587779
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Buffers: shared hit=2 read=3835
Total runtime: 364390.127 ms

With Index Only Scans:

explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Heap Fetches: 0
Buffers: shared hit=923073 read=3848
Total runtime: 4297.405 ms

As nothing is free, there are a few things to keep in mind:

* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.
* Gains will probably be smaller in real life situations, especially when data is changed between VACUUMs

This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.

==Replication improvements ==

Streaming Replication becomes more polished with this release.

One of the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources. Moreover, in case of a failover, it could be complicated to reconnect all the remaining slaves to the newly promoted master, if one is not using a tool like repmgr.

With 9.2, a standby can also send replication changes, allowing cascading replication.

Let's build this. We start with an already working 9.2 database.

We set it up for replication:

postgresql.conf:
wal_level=hot_standby #(could be archive too)
max_wal_senders=5
hot_standby=on

You'll probably also want to activate archiving in production, it won't be done here.

pg_hba.conf (do not use trust in production):
host replication replication_user 0.0.0.0/0 md5

Create the user:
create user replication_user replication password 'secret';

Clone the cluster:

pg_basebackup -h localhost -U replication_user -D data2
Password:

We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf), as both clusters are running on the same machine:
port=5433

We add a recovery.conf to tell it how to stream from the master database:
standby_mode = on
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret'

pg_ctl -D data2 start
server starting
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9D0000B8
LOG: database system is ready to accept read only connections

Now, let's add a second slave, which will use this slave:

pg_basebackup -h localhost -U replication_user -D data3 -p 5433
Password:

We edit data3's postgresql.conf to change the port:
port=5434

We modify the recovery.conf to stream from the slave:
standby_mode = on
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'

We start the third cluster:
pg_ctl -D data3 start
server starting
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9E000000
LOG: database system is ready to accept read only connections

Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).

As you may have noticed from the example, pg_basebackup now works from slaves.

There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?

pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user

will connect to the master (or a slave), and start creating files:
ls /tmp/new_logs/
00000001000000000000009E.partial

Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.

Remember to rename the last segment to remove the .partial suffix before using it with a PITR restore or any other operation.

The synchronous_commit parameter has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, and the performance improvement will be large, some people will be interested in this compromise.

==JSON datatype==

The JSON datatype is meant for storing JSON-structured data. It will validate that the input JSON string is correct JSON:

=# SELECT '{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}'::json;
json
-------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
(1 row)

=# SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json at character 8
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...
STATEMENT: SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json
LINE 1: SELECT '{"username","posts":121,"emailaddress":"john@nowhere...
^
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...

You can also convert a row type to JSON:

=#SELECT * FROM demo ;
username | posts | emailaddress
----------+-------+---------------------
john | 121 | john@nowhere.com
mickael | 215 | mickael@nowhere.com
(2 rows)

=# SELECT row_to_json(demo) FROM demo;
row_to_json
-------------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}
(2 rows)

Or an array type:

=# select array_to_json(array_agg(demo)) from demo;
array_to_json
---------------------------------------------------------------------------------------------------------------------------------------------
[{"username":"john","posts":121,"emailaddress":"john@nowhere.com"},{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}]
(1 row)

== Range Types ==

Range types are used to store a range of data of a given type. There are a few pre-defined types. They are integer (int4range), bigint (int8range), numeric (numrange), timestamp without timezone (tsrange), timestamp with timezone (tstzrange), and date (daterange).

Ranges can be made of continuous (numeric, timestamp...) or discrete (integer, date...) data types. They can be open (the bound isn't part of the range) or closed (the bound is part of the range). A bound can also be infinite.

Without these datatypes, most people solve the range problems by using two columns in a table. These range types are much more powerful, as you can use many operators on them.

Here is the intersection between then 1000(open)-2000(closed) and 1000(closed)-1200(closed) numeric range:

SELECT '(1000,2000]'::numrange * '[1000,1200]'::numrange;
?column?
-------------
(1000,1200]
(1 row)

So you can query on things like: «give me all ranges that intersect this»:

=# SELECT * from test_range ;
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-01-02 12:00:00+01"]
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(3 rows)

=# SELECT * FROM test_range WHERE period && '[2012-01-03 00:00:00,2012-01-03 12:00:00]';
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(2 rows)

This query could use an index defined like this:

=# CREATE INDEX idx_test_range on test_range USING gist (period);

You can also use these range data types to define exclusion constraints:

CREATE EXTENSION btree_gist ;
CREATE TABLE reservation (room_id int, period tstzrange);
ALTER TABLE reservation ADD EXCLUDE USING GIST (room_id WITH =, period WITH &&);

This means that now it is forbidden to have two records in this table where room_id is equal and period overlaps. The extension btree_gist is required to create a GiST index on room_id (it's an integer, it is usually indexed with a btree index).

=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (2,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');
ERROR: conflicting key value violates exclusion constraint "reservation_room_id_period_excl"
DETAIL: Key (room_id, period)=(1, ("2012-08-23 14:45:00+02","2012-08-23 15:15:00+02"))
conflicts with existing key (room_id, period)=(1, ("2012-08-23 14:00:00+02","2012-08-23 15:00:00+02")).
STATEMENT: INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');

One can also declare new range types.

=Performance improvements=

This version has performance improvements on a very large range of domains (non-exaustive):

* The most visible will probably be the Index Only Scans, which has already been introduced in this document.

* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. 

* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. 

* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption. This is especially useful on virtualized and embedded environments.

* COPY has been improved, it will generate less WAL volume and fewer locks of a table's pages. 

* Statistics are collected on array contents, allowing for better estimations of selectivity on array operations.

* Text-to-anytype concatenation and quote_literal/quote_nullable functions are not volatile any more, enabling better optimization in some cases 

* The system can now track IO durations 

This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.

If you are interested in this timing data, it's better to first check if your system will support it without too much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:

<pre>
$ pg_test_timing
Testing timing overhead for 3 seconds.
Per loop time including overhead: 28.02 nsec
Histogram of timing durations:
< usec: count percent
32: 41 0.00004%
16: 1405 0.00131%
8: 200 0.00019%
4: 388 0.00036%
2: 2982558 2.78523%
1: 104100166 97.21287%
</pre>

Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].

Anyway, here is the data you'll be able to collect if your system is ready for this:

First, you'll get per-database statistics, which will now give accurate informations about which database is doing most I/O:

<pre>
=# SELECT * FROM pg_stat_database WHERE datname = 'mydb';
-[ RECORD 1 ]--+------------------------------
datid | 16384
datname | mydb
numbackends | 1
xact_commit | 270
xact_rollback | 2
blks_read | 1961
blks_hit | 17944
tup_returned | 269035
tup_fetched | 8850
tup_inserted | 16
tup_updated | 4
tup_deleted | 45
conflicts | 0
temp_files | 0
temp_bytes | 0
deadlocks | 0
blk_read_time | 583.774
blk_write_time | 0
stats_reset | 2012-07-03 17:18:54.796817+02
</pre>
We see here that mydb has only consumed 583.774 milliseconds of read time.

Explain will benefit from this too:
<pre>
=# EXPLAIN (analyze,buffers) SELECT count(*) FROM mots ;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
Total runtime: 22.059 ms
</pre>
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).

And last, if you have enabled pg_stat_statements:
<pre>
select * from pg_stat_statements where query ~ 'words';
-[ RECORD 1 ]-------+---------------------------
userid | 10
dbid | 16384
query | select count(*) from words;
calls | 2
total_time | 78.332
rows | 2
shared_blks_hit | 0
shared_blks_read | 986
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 58.427
blk_write_time | 0
</pre>

* As for every version, the optimizer has received its share of improvements 
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it would have been efficient.

This example is straight from the developpers mailing lists :

<pre>
CREATE TABLE a (
a_id serial PRIMARY KEY NOT NULL,
b_id integer
);
CREATE INDEX a__b_id ON a USING btree (b_id);

CREATE TABLE b (
b_id serial NOT NULL,
c_id integer
);
CREATE INDEX b__c_id ON b USING btree (c_id);

CREATE TABLE c (
c_id serial PRIMARY KEY NOT NULL,
value integer UNIQUE
);

INSERT INTO b (b_id, c_id)
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);

INSERT INTO a(b_id)
SELECT g.i FROM generate_series(1, 50000) g(i);

INSERT INTO c(c_id,value)
VALUES (1,1);
</pre>

So we have a referencing b, b referencing c.

Here is an example of a query working badly with PostgreSQL 9.1:

<pre>
EXPLAIN ANALYZE SELECT 1
FROM
c
WHERE
EXISTS (
SELECT *
FROM a
JOIN b USING (b_id)
WHERE b.c_id = c.c_id)
AND c.value = 1;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)
Join Filter: (c.c_id = b.c_id)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)
Index Cond: (value = 1)
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)
Hash Cond: (a.b_id = b.b_id)
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 1954kB
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)
Total runtime: 13.842 ms
</pre>

Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.

Here's what 9.2 does with this query:

<pre>
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)
Index Cond: (value = 1)
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

The «parameterized path» is:
<pre>
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.

This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.

=SP-GiST=

SP-GiST stands for Space Partitionned GiST, GiST being Generalized Search Tree. GiST is an index type, and has been available for quite a while in PostgreSQL. GiST is already very efficient at indexing complex data types, but performance tends to suffer when the source data isn't uniformly distributed. SP-GiST tries to fix that.

As all indexing methods available in PostgreSQL, SP-GiST is a generic indexing method, meaning its purpose is to index whatever you'll throw at it, using operators you'll provide. It means that if you want to create a new datatype, and make it indexable through SP-GiST, you'll have to follow the documented API.

SP-GiST can be used to implement 3 type of indexes: trie (suffix) indexing, Quadtree (data is divided into quadrants), and k-d tree (k-dimensional tree).

For now, SP-GiST is provided with operator families called "quad_point_ops", "kd_point_ops" and "text_ops".

As their names indicate, the first one indexes point types, using a quadtree, the second one indexes point types using a k-d tree, and the third one indexes text, using suffix.

=pg_stat_statements=

This contrib module has received a lot of improvements in this version:

* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.

<pre>
=#SELECT * FROM words WHERE word= 'foo';
word
------
(0 ligne)

=# SELECT * FROM words WHERE word= 'bar';
word
------
bar

=#select * from pg_stat_statements where query like '%words where%';
-[ RECORD 1 ]-------+-----------------------------------
userid | 10
dbid | 16384
query | SELECT * FROM words WHERE word= ?;
calls | 2
total_time | 142.314
rows | 1
shared_blks_hit | 3
shared_blks_read | 5
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 142.165
blk_write_time | 0

</pre>

The two queries are shown as one in pg_stat_statements.

* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That makes it is easier to interpret, and avoids the double-counting there was with PostgreSQL 9.1.

* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.

= Explain improvements=

* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive 

=# EXPLAIN (analyze on,timing off) SELECT * FROM reservation ;
QUERY PLAN
----------------------------------------------------------------------------------------
Seq Scan on reservation (cost=0.00..22.30 rows=1230 width=36) (actual rows=2 loops=1)
Total runtime: 0.045 ms

* Have EXPLAIN ANALYZE report the number of rows rejected by filter steps 

This new feature makes it much easier to know how many rows are removed by a filter (and spot potential places to put indexes):

=# EXPLAIN ANALYZE SELECT * FROM test WHERE a ~ 'tra';
QUERY PLAN
---------------------------------------------------------------------------------------------------------------
Seq Scan on test (cost=0.00..106876.56 rows=2002 width=11) (actual time=2.914..8538.285 rows=120256 loops=1)
Filter: (a ~ 'tra'::text)
Rows Removed by Filter: 5905600
Total runtime: 8549.539 ms
(4 rows)

=Backward compatibility=

These changes may incur regressions in your applications.

==Ensure that xpath() escapes special characters in string values  ==

Before 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
<

'<' Isn't valid XML.
</pre>
With 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
&lt;
</pre>

==Remove hstore's => operator ==
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.

In 9.1:
<pre>
=# SELECT 'a'=>'b';
?column?
----------
"a"=>"b"
(1 row)

=# SELECT pg_typeof('a'=>'b');
pg_typeof
-----------
hstore
(1 row)
</pre>

With 9.2:
<pre>
SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown at character 11
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
STATEMENT: SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown
LINE 1: SELECT 'a'=>'b';
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
</pre>

It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:

<pre>
=# select hstore('a=>b');
hstore
----------
"a"=>"b"
(1 row)

=# select hstore('a','b');
hstore
----------
"a"=>"b"
(1 row)
</pre>
are still two valid ways to input a hstore.

"=>" is removed as an operator as it is a reserved keyword in SQL.

==Have pg_relation_size() and friends return NULL if the object does not exist ==

A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.

==Remove the spclocation field from pg_tablespace ==

The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.

So this field has been removed. To get the tablespace's location, use pg_tablespace_location():

<pre>
=# SELECT *, pg_tablespace_location(oid) AS spclocation FROM pg_tablespace;
spcname | spcowner | spcacl | spcoptions | spclocation
------------+----------+--------+------------+----------------
pg_default | 10 | | |
pg_global | 10 | | |
tmptblspc | 10 | | | /tmp/tmptblspc
</pre>

==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight ==

With PostgreSQL 9.1:

<pre>
=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341180000
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>

There is no difference in behaviour between a timstamp with or without timezone.

With 9.2:
<pre>
=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341187200
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.

==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 ==

The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.

Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.

With 9.1:
<pre>
=# SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
1200-07-02
</pre>

With 9.2:
<pre>
SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
2200-07-02
</pre>

==pg_stat_activity and pg_stat_replication's definitions have changed ==

The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:

* current_query disappears and is replaced by two columns:
** state: is the session running a query, waiting
** query: what is the last run (or still running if stat is "active") query
* The column procpid is renamed to pid, to be consistent with other system views

The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.

Here is an example:
<pre>
-[ RECORD 1 ]----+---------------------------------
datid | 16384
datname | postgres
pid | 20804
usesysid | 10
usename | postgres
application_name | psql
client_addr |
client_hostname |
client_port | -1
backend_start | 2012-07-02 15:02:51.146427+02
xact_start | 2012-07-02 15:15:28.386865+02
query_start | 2012-07-02 15:15:30.410834+02
state_change | 2012-07-02 15:15:30.411287+02
waiting | f
state | idle in transaction
query | DELETE FROM test;
</pre>

With PostgreSQL 9.1, all we would have would be «idle in transaction».

As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.
The view pg_stat_replication has also changed. The column procpid is renamed to pid, to also be consistent with other system views.

==Change all SQL-level statistics timing values to float8-stored milliseconds ==

pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.

==postgresql.conf parameters changes ==

* silent_mode has been removed. Use pg_ctl -l postmaster.log
* wal_sender_delay has been removed. It is no longer needed
* custom_variable_classes has been removed. All «classes» are accepted without declaration now
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files

= Other new features =

== DROP INDEX CONCURRENTLY ==

The regular DROP INDEX command takes an exclusive lock on the table. Most of the time, this isn't a problem, because this lock is short-lived. The problem usually occurs when:

* A long-running transaction is running, and has a (shared) lock on the table
* A DROP INDEX is run on this table in another session, asking for an exclusive lock (and waiting for it, as it won't be granted until the long-running transaction ends)

At this point, all other transactions needing to take a shared lock on the table (for a simple SELECT for instance) will have to wait too: their lock acquisition is queued after the DROP INDEX's one.

DROP INDEX CONCURRENTLY works around this and won't lock normal DML statements, just as CREATE INDEX CONCURRENTLY. The limitations are also the same: Since you can only DROP one index with the CONCURRENTLY option, and the CASCADE option is not supported.

== NOT VALID CHECK constraints ==

PostgreSQL 9.1 introduced «NOT VALID» foreign keys. This has been extended to CHECK constraints. Adding a «NOT VALID» constraint on a table means that current data won't be validated, only new and updated rows.

=# CREATE TABLE test (a int);
CREATE TABLE
=# INSERT INTO test SELECT generate_series(1,100);
INSERT 0 100
=# ALTER TABLE test ADD CHECK (a>100) NOT VALID;
ALTER TABLE
=# INSERT INTO test VALUES (99);
ERROR: new row for relation "test" violates check constraint "test_a_check"
DETAIL: Failing row contains (99).
=# INSERT INTO test VALUES (101);
INSERT 0 1

Then, later, we can validate the whole table:

=# ALTER TABLE test VALIDATE CONSTRAINT test_a_check ;
ERROR: check constraint "test_a_check" is violated by some row

Domains, which are types with added constraints, can also be declared as not valid, and validated later.

Check constraints can also be renamed now:

=# ALTER TABLE test RENAME CONSTRAINT test_a_check TO validate_a;
ALTER TABLE

== NO INHERIT constraints ==

Here is another improvement about constraints: they can be declared as not inheritable, which will be useful in partitioned environments. Let's take PostgreSQL documentation example, and see how it improves the situation:

CREATE TABLE measurement (
city_id int not null,
logdate date not null,
peaktemp int,
unitsales int,
CHECK (logdate IS NULL) NO INHERIT
);

CREATE TABLE measurement_y2006m02 (
CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
) INHERITS (measurement);
CREATE TABLE measurement_y2006m03 (
CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' )
) INHERITS (measurement);

INSERT INTO measurement VALUES (1,'2006-02-20',1,1);
ERROR: new row for relation "measurement" violates check constraint "measurement_logdate_check"
DETAIL: Failing row contains (1, 2006-02-20, 1, 1).
INSERT INTO measurement_y2006m02 VALUES (1,'2006-02-20',1,1);
INSERT 0 1

Until now, every check constraint created on measurement would have been inherited by children tables. So adding a constraint forbidding inserts, or allowing only some of them, on the parent table was impossible.

== Reduce ALTER TABLE rewrites ==

A table won't get rewritten anymore during an ALTER TABLE when changing the type of a column in the following cases:

* varchar(x) to varchar(y) when y>=x. It works too if going from varchar(x) to varchar or text (no size limitation)
* numeric(x,z) to numeric(y,z) when y>=x, or to numeric without specifier
* varbit(x) to varbit(y) when y>=x, or to varbit without specifier
* timestamp(x) to timestamp(y) when y>=x or timestamp without specifier
* timestamptz(x) to timestamptz(y) when y>=x or timestamptz without specifier
* interval(x) to interval(y) when y>=x or interval without specifier

== Security barriers and Leakproof ==

This new feature has to do with views security. First, let's explain the problem, with a very simplified example:

=# CREATE TABLE all_data (company_id int, company_data varchar);
CREATE TABLE
=# INSERT INTO all_data VALUES (1,'secret_data_for_company_1');
INSERT 0 1
=# INSERT INTO all_data VALUES (2,'secret_data_for_company_2');
INSERT 0 1
=# CREATE VIEW company1_data AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW

This is a quite classical way of giving access to only a part of a table to a user: we'll create a user for company_id 1, grant to him the right to access company1_data, and deny him the right to access all_data.

The plan to this query is the following:

=# explain SELECT * FROM company1_data ;
QUERY PLAN
----------------------------------------------------------
Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)

Even if there was more data, a sequential scan could still be forced: just "SET enable_indexscan to OFF" and the likes.

So this query reads all the records from all_data, filters them, and returns to the user only the matching rows. There is a way to display scanned records before they are filtered: just create a function with a very low cost, and call it while doing the query:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

This function just has to cost less than the = operator, which costs 1, to be executed first.

The result is this:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

We got access to the record from the second company (in the NOTICE messages).

So this is the first new feature: the view can be declared as implementing "security barriers":

=# CREATE VIEW company1_data WITH (security_barrier) AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

The view is not leaking anymore. The problem, of course, is that there is a performance impact: maybe the "peek" function could have made the query faster, by filtering lots of rows early in the plan. The rows are now filtered in two separate plan steps:

=# explain SELECT * FROM company1_data WHERE peek(company1_data.company_data);
QUERY PLAN
--------------------------------------------------------------------
Subquery Scan on company1_data (cost=0.00..25.44 rows=2 width=36)
Filter: peek((company1_data.company_data)::text)
-> Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)

This leads to the complementary feature: some function may be declared as "LEAKPROOF", meaning that they won't leak the data they are passed into error or notice messages.

Declaring our peek function as LEAKPROOF is a very bad idea, but let's do it just to demonstrate how it's used:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LEAKPROOF LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

A LEAKPROOF function is executed «normally»:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

Of course, in our case, peek isn't LEAKPROOF and shouldn't be declared as such. Only superuser have the permission to declare a LEAKPROOF function.

== New options for pg_dump ==

Until now, one could ask pg_dump to dump a table's data, or a table's meta-data (DDL statements for creating the table's structure, indexes, constraints). Some meta-data is better restored before the data (the table's structure, check constraints), some is better after the data (indexes, unique constraints, foreign keys…), for performance reasons mostly.

So there are now a few more options:

* --section=pre-data: dump what's needed before restoring the data. Of course, this can be combined with a -t for instance, to specify only one table
* --section=post-data : dump what's needed after restoring the data.
* --section=data: dump the data
* --exclude-table-data: dump everything, except THIS table's data. It means pg_dump will still dump other tables' data.

[[Category:PostgreSQL 9.2]]

What's new in PostgreSQL 9.2

2012-09-25T12:40:42Z

Intgr: /* Security barriers and Leakproof */ unnecessary

{{Languages}}

This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release – PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.

=Major new features=

==Index-only scans ==

In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.

It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.

There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.

The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).

If the data has been read only since the last VACUUM then the data is All Visible and the index only scan feature can improve performance.

Here is an example.

CREATE TABLE demo_ios (col1 float, col2 float, col3 text);

In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains won't be that big in real life.

INSERT INTO demo_ios SELECT generate_series(1,100000000),random(), 'mynotsolongstring';

SELECT pg_size_pretty(pg_total_relation_size('demo_ios'));
pg_size_pretty
----------------
6512 MB

Let's pretend that the query is this:

SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.01 AND 0.02

In order to use an index only scan on this query, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).

CREATE index idx_demo_ios on demo_ios(col2,col1);

We vacuum the table, so that the visibility map to be up-to-date:

VACUUM demo_ios;

All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).

Let's first try without Index Only Scans:

SET enable_indexonlyscan to off;

EXPLAIN (analyze,buffers) select col1,col2 FROM demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Rows Removed by Index Recheck: 68098621
Buffers: shared hit=2 read=587779
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Buffers: shared hit=2 read=3835
Total runtime: 364390.127 ms

With Index Only Scans:

explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Heap Fetches: 0
Buffers: shared hit=923073 read=3848
Total runtime: 4297.405 ms

As nothing is free, there are a few things to keep in mind:

* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.
* Gains will probably be smaller in real life situations, especially when data is changed between VACUUMs

This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.

==Replication improvements ==

Streaming Replication becomes more polished with this release.

One of the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources. Moreover, in case of a failover, it could be complicated to reconnect all the remaining slaves to the newly promoted master, if one is not using a tool like repmgr.

With 9.2, a standby can also send replication changes, allowing cascading replication.

Let's build this. We start with an already working 9.2 database.

We set it up for replication:

postgresql.conf:
wal_level=hot_standby #(could be archive too)
max_wal_senders=5
hot_standby=on

You'll probably also want to activate archiving in production, it won't be done here.

pg_hba.conf (do not use trust in production):
host replication replication_user 0.0.0.0/0 md5

Create the user:
create user replication_user replication password 'secret';

Clone the cluster:

pg_basebackup -h localhost -U replication_user -D data2
Password:

We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf), as both clusters are running on the same machine:
port=5433

We add a recovery.conf to tell it how to stream from the master database:
standby_mode = on
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret'

pg_ctl -D data2 start
server starting
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9D0000B8
LOG: database system is ready to accept read only connections

Now, let's add a second slave, which will use this slave:

pg_basebackup -h localhost -U replication_user -D data3 -p 5433
Password:

We edit data3's postgresql.conf to change the port:
port=5434

We modify the recovery.conf to stream from the slave:
standby_mode = on
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'

We start the third cluster:
pg_ctl -D data3 start
server starting
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9E000000
LOG: database system is ready to accept read only connections

Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).

As you may have noticed from the example, pg_basebackup now works from slaves.

There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?

pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user

will connect to the master (or a slave), and start creating files:
ls /tmp/new_logs/
00000001000000000000009E.partial

Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.

Remember to rename the last segment to remove the .partial suffix before using it with a PITR restore or any other operation.

The synchronous_commit parameter has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, and the performance improvement will be large, some people will be interested in this compromise.

==JSON datatype==

The JSON datatype is meant for storing JSON-structured data. It will validate that the input JSON string is correct JSON:

=# SELECT '{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}'::json;
json
-------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
(1 row)

=# SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json at character 8
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...
STATEMENT: SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json
LINE 1: SELECT '{"username","posts":121,"emailaddress":"john@nowhere...
^
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...

You can also convert a row type to JSON:

=#SELECT * FROM demo ;
username | posts | emailaddress
----------+-------+---------------------
john | 121 | john@nowhere.com
mickael | 215 | mickael@nowhere.com
(2 rows)

=# SELECT row_to_json(demo) FROM demo;
row_to_json
-------------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}
(2 rows)

Or an array type:

=# select array_to_json(array_agg(demo)) from demo;
array_to_json
---------------------------------------------------------------------------------------------------------------------------------------------
[{"username":"john","posts":121,"emailaddress":"john@nowhere.com"},{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}]
(1 row)

== Range Types ==

Range types are used to store a range of data of a given type. There are a few pre-defined types. They are integer (int4range), bigint (int8range), numeric (numrange), timestamp without timezone (tsrange), timestamp with timezone (tstzrange), and date (daterange).

Ranges can be made of continuous (numeric, timestamp...) or discrete (integer, date...) data types. They can be open (the bound isn't part of the range) or closed (the bound is part of the range). A bound can also be infinite.

Without these datatypes, most people solve the range problems by using two columns in a table. These range types are much more powerful, as you can use many operators on them.

Here is the intersection between then 1000(open)-2000(closed) and 1000(closed)-1200(closed) numeric range:

SELECT '(1000,2000]'::numrange * '[1000,1200]'::numrange;
?column?
-------------
(1000,1200]
(1 row)

So you can query on things like: «give me all ranges that intersect this»:

=# SELECT * from test_range ;
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-01-02 12:00:00+01"]
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(3 rows)

=# SELECT * FROM test_range WHERE period && '[2012-01-03 00:00:00,2012-01-03 12:00:00]';
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(2 rows)

This query could use an index defined like this:

=# CREATE INDEX idx_test_range on test_range USING gist (period);

You can also use these range data types to define exclusion constraints:

CREATE EXTENSION btree_gist ;
CREATE TABLE reservation (room_id int, period tstzrange);
ALTER TABLE reservation ADD EXCLUDE USING GIST (room_id WITH =, period WITH &&);

This means that now it is forbidden to have two records in this table where room_id is equal and period overlaps. The extension btree_gist is required to create a GiST index on room_id (it's an integer, it is usually indexed with a btree index).

=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (2,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');
ERROR: conflicting key value violates exclusion constraint "reservation_room_id_period_excl"
DETAIL: Key (room_id, period)=(1, ("2012-08-23 14:45:00+02","2012-08-23 15:15:00+02"))
conflicts with existing key (room_id, period)=(1, ("2012-08-23 14:00:00+02","2012-08-23 15:00:00+02")).
STATEMENT: INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');

One can also declare new range types.

=Performance improvements=

This version has performance improvements on a very large range of domains (non-exaustive):

* The most visible will probably be the Index Only Scans, which has already been introduced in this document.

* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. 

* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. 

* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption. This is especially useful on virtualized and embedded environments.

* COPY has been improved, it will generate less WAL volume and fewer locks of a table's pages. 

* Statistics are collected on array contents, allowing for better estimations of selectivity on array operations.

* Text-to-anytype concatenation and quote_literal/quote_nullable functions are not volatile any more, enabling better optimization in some cases 

* The system can now track IO durations 

This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.

If you are interested in this timing data, it's better to first check if your system will support it without too much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:

<pre>
$ pg_test_timing
Testing timing overhead for 3 seconds.
Per loop time including overhead: 28.02 nsec
Histogram of timing durations:
< usec: count percent
32: 41 0.00004%
16: 1405 0.00131%
8: 200 0.00019%
4: 388 0.00036%
2: 2982558 2.78523%
1: 104100166 97.21287%
</pre>

Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].

Anyway, here is the data you'll be able to collect if your system is ready for this:

First, you'll get per-database statistics, which will now give accurate informations about which database is doing most I/O:

<pre>
=# SELECT * FROM pg_stat_database WHERE datname = 'mydb';
-[ RECORD 1 ]--+------------------------------
datid | 16384
datname | mydb
numbackends | 1
xact_commit | 270
xact_rollback | 2
blks_read | 1961
blks_hit | 17944
tup_returned | 269035
tup_fetched | 8850
tup_inserted | 16
tup_updated | 4
tup_deleted | 45
conflicts | 0
temp_files | 0
temp_bytes | 0
deadlocks | 0
blk_read_time | 583.774
blk_write_time | 0
stats_reset | 2012-07-03 17:18:54.796817+02
</pre>
We see here that mydb has only consumed 583.774 milliseconds of read time.

Explain will benefit from this too:
<pre>
=# EXPLAIN (analyze,buffers) SELECT count(*) FROM mots ;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
Total runtime: 22.059 ms
</pre>
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).

And last, if you have enabled pg_stat_statements:
<pre>
select * from pg_stat_statements where query ~ 'words';
-[ RECORD 1 ]-------+---------------------------
userid | 10
dbid | 16384
query | select count(*) from words;
calls | 2
total_time | 78.332
rows | 2
shared_blks_hit | 0
shared_blks_read | 986
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 58.427
blk_write_time | 0
</pre>

* As for every version, the optimizer has received its share of improvements 
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it would have been efficient.

This example is straight from the developpers mailing lists :

<pre>
CREATE TABLE a (
a_id serial PRIMARY KEY NOT NULL,
b_id integer
);
CREATE INDEX a__b_id ON a USING btree (b_id);

CREATE TABLE b (
b_id serial NOT NULL,
c_id integer
);
CREATE INDEX b__c_id ON b USING btree (c_id);

CREATE TABLE c (
c_id serial PRIMARY KEY NOT NULL,
value integer UNIQUE
);

INSERT INTO b (b_id, c_id)
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);

INSERT INTO a(b_id)
SELECT g.i FROM generate_series(1, 50000) g(i);

INSERT INTO c(c_id,value)
VALUES (1,1);
</pre>

So we have a referencing b, b referencing c.

Here is an example of a query working badly with PostgreSQL 9.1:

<pre>
EXPLAIN ANALYZE SELECT 1
FROM
c
WHERE
EXISTS (
SELECT *
FROM a
JOIN b USING (b_id)
WHERE b.c_id = c.c_id)
AND c.value = 1;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)
Join Filter: (c.c_id = b.c_id)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)
Index Cond: (value = 1)
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)
Hash Cond: (a.b_id = b.b_id)
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 1954kB
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)
Total runtime: 13.842 ms
</pre>

Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.

Here's what 9.2 does with this query:

<pre>
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)
Index Cond: (value = 1)
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

The «parameterized path» is:
<pre>
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.

This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.

=SP-GiST=

SP-GiST stands for Space Partitionned GiST, GiST being Generalized Search Tree. GiST is an index type, and has been available for quite a while in PostgreSQL. GiST is already very efficient at indexing complex data types, but performance tends to suffer when the source data isn't uniformly distributed. SP-GiST tries to fix that.

As all indexing methods available in PostgreSQL, SP-GiST is a generic indexing method, meaning its purpose is to index whatever you'll throw at it, using operators you'll provide. It means that if you want to create a new datatype, and make it indexable through SP-GiST, you'll have to follow the documented API.

SP-GiST can be used to implement 3 type of indexes: trie (suffix) indexing, Quadtree (data is divided into quadrants), and k-d tree (k-dimensional tree).

For now, SP-GiST is provided with operator families called "quad_point_ops", "kd_point_ops" and "text_ops".

As their names indicate, the first one indexes point types, using a quadtree, the second one indexes point types using a k-d tree, and the third one indexes text, using suffix.

=pg_stat_statements=

This contrib module has received a lot of improvements in this version:

* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.

<pre>
=#SELECT * FROM words WHERE word= 'foo';
word
------
(0 ligne)

=# SELECT * FROM words WHERE word= 'bar';
word
------
bar

=#select * from pg_stat_statements where query like '%words where%';
-[ RECORD 1 ]-------+-----------------------------------
userid | 10
dbid | 16384
query | SELECT * FROM words WHERE word= ?;
calls | 2
total_time | 142.314
rows | 1
shared_blks_hit | 3
shared_blks_read | 5
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 142.165
blk_write_time | 0

</pre>

The two queries are shown as one in pg_stat_statements.

* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That makes it is easier to interpret, and avoids the double-counting there was with PostgreSQL 9.1.

* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.

= Explain improvements=

* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive 

=# EXPLAIN (analyze on,timing off) SELECT * FROM reservation ;
QUERY PLAN
----------------------------------------------------------------------------------------
Seq Scan on reservation (cost=0.00..22.30 rows=1230 width=36) (actual rows=2 loops=1)
Total runtime: 0.045 ms

* Have EXPLAIN ANALYZE report the number of rows rejected by filter steps 

This new feature makes it much easier to know how many rows are removed by a filter (and spot potential places to put indexes):

=# EXPLAIN ANALYZE SELECT * FROM test WHERE a ~ 'tra';
QUERY PLAN
---------------------------------------------------------------------------------------------------------------
Seq Scan on test (cost=0.00..106876.56 rows=2002 width=11) (actual time=2.914..8538.285 rows=120256 loops=1)
Filter: (a ~ 'tra'::text)
Rows Removed by Filter: 5905600
Total runtime: 8549.539 ms
(4 rows)

=Backward compatibility=

These changes may incur regressions in your applications.

==Ensure that xpath() escapes special characters in string values  ==

Before 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
<

'<' Isn't valid XML.
</pre>
With 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
&lt;
</pre>

==Remove hstore's => operator ==
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.

In 9.1:
<pre>
=# SELECT 'a'=>'b';
?column?
----------
"a"=>"b"
(1 row)

=# SELECT pg_typeof('a'=>'b');
pg_typeof
-----------
hstore
(1 row)
</pre>

With 9.2:
<pre>
SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown at character 11
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
STATEMENT: SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown
LINE 1: SELECT 'a'=>'b';
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
</pre>

It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:

<pre>
=# select hstore('a=>b');
hstore
----------
"a"=>"b"
(1 row)

=# select hstore('a','b');
hstore
----------
"a"=>"b"
(1 row)
</pre>
are still two valid ways to input a hstore.

"=>" is removed as an operator as it is a reserved keyword in SQL.

==Have pg_relation_size() and friends return NULL if the object does not exist ==

A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.

==Remove the spclocation field from pg_tablespace ==

The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.

So this field has been removed. To get the tablespace's location, use pg_tablespace_location():

<pre>
=# SELECT *, pg_tablespace_location(oid) AS spclocation FROM pg_tablespace;
spcname | spcowner | spcacl | spcoptions | spclocation
------------+----------+--------+------------+----------------
pg_default | 10 | | |
pg_global | 10 | | |
tmptblspc | 10 | | | /tmp/tmptblspc
</pre>

==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight ==

With PostgreSQL 9.1:

<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341180000
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>

There is no difference in behaviour between a timstamp with or without timezone.

With 9.2:
<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341187200
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.

==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 ==

The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.

Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.

With 9.1:
<pre>
=# SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
1200-07-02
</pre>

With 9.2:
<pre>
SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
2200-07-02
</pre>

==pg_stat_activity and pg_stat_replication's definitions have changed ==

The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:

* current_query disappears and is replaced by two columns:
** state: is the session running a query, waiting
** query: what is the last run (or still running if stat is "active") query
* The column procpid is renamed to pid, to be consistent with other system views

The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.

Here is an example:
<pre>
-[ RECORD 1 ]----+---------------------------------
datid | 16384
datname | postgres
pid | 20804
usesysid | 10
usename | postgres
application_name | psql
client_addr |
client_hostname |
client_port | -1
backend_start | 2012-07-02 15:02:51.146427+02
xact_start | 2012-07-02 15:15:28.386865+02
query_start | 2012-07-02 15:15:30.410834+02
state_change | 2012-07-02 15:15:30.411287+02
waiting | f
state | idle in transaction
query | DELETE FROM test;
</pre>

With PostgreSQL 9.1, all we would have would be «idle in transaction».

As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.
The view pg_stat_replication has also changed. The column procpid is renamed to pid, to also be consistent with other system views.

==Change all SQL-level statistics timing values to float8-stored milliseconds ==

pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.

==postgresql.conf parameters changes ==

* silent_mode has been removed. Use pg_ctl -l postmaster.log
* wal_sender_delay has been removed. It is no longer needed
* custom_variable_classes has been removed. All «classes» are accepted without declaration now
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files

= Other new features =

== DROP INDEX CONCURRENTLY ==

The regular DROP INDEX command takes an exclusive lock on the table. Most of the time, this isn't a problem, because this lock is short-lived. The problem usually occurs when:

* A long-running transaction is running, and has a (shared) lock on the table
* A DROP INDEX is run on this table in another session, asking for an exclusive lock (and waiting for it, as it won't be granted until the long-running transaction ends)

At this point, all other transactions needing to take a shared lock on the table (for a simple SELECT for instance) will have to wait too: their lock acquisition is queued after the DROP INDEX's one.

DROP INDEX CONCURRENTLY works around this and won't lock normal DML statements, just as CREATE INDEX CONCURRENTLY. The limitations are also the same: Since you can only DROP one index with the CONCURRENTLY option, and the CASCADE option is not supported.

== NOT VALID CHECK constraints ==

PostgreSQL 9.1 introduced «NOT VALID» foreign keys. This has been extended to CHECK constraints. Adding a «NOT VALID» constraint on a table means that current data won't be validated, only new and updated rows.

=# CREATE TABLE test (a int);
CREATE TABLE
=# INSERT INTO test SELECT generate_series(1,100);
INSERT 0 100
=# ALTER TABLE test ADD CHECK (a>100) NOT VALID;
ALTER TABLE
=# INSERT INTO test VALUES (99);
ERROR: new row for relation "test" violates check constraint "test_a_check"
DETAIL: Failing row contains (99).
=# INSERT INTO test VALUES (101);
INSERT 0 1

Then, later, we can validate the whole table:

=# ALTER TABLE test VALIDATE CONSTRAINT test_a_check ;
ERROR: check constraint "test_a_check" is violated by some row

Domains, which are types with added constraints, can also be declared as not valid, and validated later.

Check constraints can also be renamed now:

=# ALTER TABLE test RENAME CONSTRAINT test_a_check TO validate_a;
ALTER TABLE

== NO INHERIT constraints ==

Here is another improvement about constraints: they can be declared as not inheritable, which will be useful in partitioned environments. Let's take PostgreSQL documentation example, and see how it improves the situation:

CREATE TABLE measurement (
city_id int not null,
logdate date not null,
peaktemp int,
unitsales int,
CHECK (logdate IS NULL) NO INHERIT
);

CREATE TABLE measurement_y2006m02 (
CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
) INHERITS (measurement);
CREATE TABLE measurement_y2006m03 (
CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' )
) INHERITS (measurement);

INSERT INTO measurement VALUES (1,'2006-02-20',1,1);
ERROR: new row for relation "measurement" violates check constraint "measurement_logdate_check"
DETAIL: Failing row contains (1, 2006-02-20, 1, 1).
INSERT INTO measurement_y2006m02 VALUES (1,'2006-02-20',1,1);
INSERT 0 1

Until now, every check constraint created on measurement would have been inherited by children tables. So adding a constraint forbidding inserts, or allowing only some of them, on the parent table was impossible.

== Reduce ALTER TABLE rewrites ==

A table won't get rewritten anymore during an ALTER TABLE when changing the type of a column in the following cases:

* varchar(x) to varchar(y) when y>=x. It works too if going from varchar(x) to varchar or text (no size limitation)
* numeric(x,z) to numeric(y,z) when y>=x, or to numeric without specifier
* varbit(x) to varbit(y) when y>=x, or to varbit without specifier
* timestamp(x) to timestamp(y) when y>=x or timestamp without specifier
* timestamptz(x) to timestamptz(y) when y>=x or timestamptz without specifier
* interval(x) to interval(y) when y>=x or interval without specifier

== Security barriers and Leakproof ==

This new feature has to do with views security. First, let's explain the problem, with a very simplified example:

=# CREATE TABLE all_data (company_id int, company_data varchar);
CREATE TABLE
=# INSERT INTO all_data VALUES (1,'secret_data_for_company_1');
INSERT 0 1
=# INSERT INTO all_data VALUES (2,'secret_data_for_company_2');
INSERT 0 1
=# CREATE VIEW company1_data AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW

This is a quite classical way of giving access to only a part of a table to a user: we'll create a user for company_id 1, grant to him the right to access company1_data, and deny him the right to access all_data.

The plan to this query is the following:

=# explain SELECT * FROM company1_data ;
QUERY PLAN
----------------------------------------------------------
Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)

Even if there was more data, a sequential scan could still be forced: just "SET enable_indexscan to OFF" and the likes.

So this query reads all the records from all_data, filters them, and returns to the user only the matching rows. There is a way to display scanned records before they are filtered: just create a function with a very low cost, and call it while doing the query:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

This function just has to cost less than the = operator, which costs 1, to be executed first.

The result is this:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

We got access to the record from the second company (in the NOTICE messages).

So this is the first new feature: the view can be declared as implementing "security barriers":

=# CREATE VIEW company1_data WITH (security_barrier) AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

The view is not leaking anymore. The problem, of course, is that there is a performance impact: maybe the "peek" function could have made the query faster, by filtering lots of rows early in the plan. The rows are now filtered in two separate plan steps:

=# explain SELECT * FROM company1_data WHERE peek(company1_data.company_data);
QUERY PLAN
--------------------------------------------------------------------
Subquery Scan on company1_data (cost=0.00..25.44 rows=2 width=36)
Filter: peek((company1_data.company_data)::text)
-> Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)

This leads to the complementary feature: some function may be declared as "LEAKPROOF", meaning that they won't leak the data they are passed into error or notice messages.

Declaring our peek function as LEAKPROOF is a very bad idea, but let's do it just to demonstrate how it's used:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LEAKPROOF LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

A LEAKPROOF function is executed «normally»:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

Of course, in our case, peek isn't LEAKPROOF and shouldn't be declared as such. Only superuser have the permission to declare a LEAKPROOF function.

== New options for pg_dump ==

Until now, one could ask pg_dump to dump a table's data, or a table's meta-data (DDL statements for creating the table's structure, indexes, constraints). Some meta-data is better restored before the data (the table's structure, check constraints), some is better after the data (indexes, unique constraints, foreign keys…), for performance reasons mostly.

So there are now a few more options:

* --section=pre-data: dump what's needed before restoring the data. Of course, this can be combined with a -t for instance, to specify only one table
* --section=post-data : dump what's needed after restoring the data.
* --section=data: dump the data
* --exclude-table-data: dump everything, except THIS table's data. It means pg_dump will still dump other tables' data.

[[Category:PostgreSQL 9.2]]

What's new in PostgreSQL 9.2

2012-09-25T12:40:10Z

Intgr: /* Security barriers and Leakproof */ Add example plan with security_barrier views

{{Languages}}

This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release – PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.

=Major new features=

==Index-only scans ==

In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.

It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.

There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.

The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).

If the data has been read only since the last VACUUM then the data is All Visible and the index only scan feature can improve performance.

Here is an example.

CREATE TABLE demo_ios (col1 float, col2 float, col3 text);

In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains won't be that big in real life.

INSERT INTO demo_ios SELECT generate_series(1,100000000),random(), 'mynotsolongstring';

SELECT pg_size_pretty(pg_total_relation_size('demo_ios'));
pg_size_pretty
----------------
6512 MB

Let's pretend that the query is this:

SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.01 AND 0.02

In order to use an index only scan on this query, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).

CREATE index idx_demo_ios on demo_ios(col2,col1);

We vacuum the table, so that the visibility map to be up-to-date:

VACUUM demo_ios;

All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).

Let's first try without Index Only Scans:

SET enable_indexonlyscan to off;

EXPLAIN (analyze,buffers) select col1,col2 FROM demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Rows Removed by Index Recheck: 68098621
Buffers: shared hit=2 read=587779
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Buffers: shared hit=2 read=3835
Total runtime: 364390.127 ms

With Index Only Scans:

explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Heap Fetches: 0
Buffers: shared hit=923073 read=3848
Total runtime: 4297.405 ms

As nothing is free, there are a few things to keep in mind:

* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.
* Gains will probably be smaller in real life situations, especially when data is changed between VACUUMs

This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.

==Replication improvements ==

Streaming Replication becomes more polished with this release.

One of the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources. Moreover, in case of a failover, it could be complicated to reconnect all the remaining slaves to the newly promoted master, if one is not using a tool like repmgr.

With 9.2, a standby can also send replication changes, allowing cascading replication.

Let's build this. We start with an already working 9.2 database.

We set it up for replication:

postgresql.conf:
wal_level=hot_standby #(could be archive too)
max_wal_senders=5
hot_standby=on

You'll probably also want to activate archiving in production, it won't be done here.

pg_hba.conf (do not use trust in production):
host replication replication_user 0.0.0.0/0 md5

Create the user:
create user replication_user replication password 'secret';

Clone the cluster:

pg_basebackup -h localhost -U replication_user -D data2
Password:

We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf), as both clusters are running on the same machine:
port=5433

We add a recovery.conf to tell it how to stream from the master database:
standby_mode = on
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret'

pg_ctl -D data2 start
server starting
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9D0000B8
LOG: database system is ready to accept read only connections

Now, let's add a second slave, which will use this slave:

pg_basebackup -h localhost -U replication_user -D data3 -p 5433
Password:

We edit data3's postgresql.conf to change the port:
port=5434

We modify the recovery.conf to stream from the slave:
standby_mode = on
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'

We start the third cluster:
pg_ctl -D data3 start
server starting
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9E000000
LOG: database system is ready to accept read only connections

Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).

As you may have noticed from the example, pg_basebackup now works from slaves.

There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?

pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user

will connect to the master (or a slave), and start creating files:
ls /tmp/new_logs/
00000001000000000000009E.partial

Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.

Remember to rename the last segment to remove the .partial suffix before using it with a PITR restore or any other operation.

The synchronous_commit parameter has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, and the performance improvement will be large, some people will be interested in this compromise.

==JSON datatype==

The JSON datatype is meant for storing JSON-structured data. It will validate that the input JSON string is correct JSON:

=# SELECT '{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}'::json;
json
-------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
(1 row)

=# SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json at character 8
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...
STATEMENT: SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json
LINE 1: SELECT '{"username","posts":121,"emailaddress":"john@nowhere...
^
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...

You can also convert a row type to JSON:

=#SELECT * FROM demo ;
username | posts | emailaddress
----------+-------+---------------------
john | 121 | john@nowhere.com
mickael | 215 | mickael@nowhere.com
(2 rows)

=# SELECT row_to_json(demo) FROM demo;
row_to_json
-------------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}
(2 rows)

Or an array type:

=# select array_to_json(array_agg(demo)) from demo;
array_to_json
---------------------------------------------------------------------------------------------------------------------------------------------
[{"username":"john","posts":121,"emailaddress":"john@nowhere.com"},{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}]
(1 row)

== Range Types ==

Range types are used to store a range of data of a given type. There are a few pre-defined types. They are integer (int4range), bigint (int8range), numeric (numrange), timestamp without timezone (tsrange), timestamp with timezone (tstzrange), and date (daterange).

Ranges can be made of continuous (numeric, timestamp...) or discrete (integer, date...) data types. They can be open (the bound isn't part of the range) or closed (the bound is part of the range). A bound can also be infinite.

Without these datatypes, most people solve the range problems by using two columns in a table. These range types are much more powerful, as you can use many operators on them.

Here is the intersection between then 1000(open)-2000(closed) and 1000(closed)-1200(closed) numeric range:

SELECT '(1000,2000]'::numrange * '[1000,1200]'::numrange;
?column?
-------------
(1000,1200]
(1 row)

So you can query on things like: «give me all ranges that intersect this»:

=# SELECT * from test_range ;
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-01-02 12:00:00+01"]
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(3 rows)

=# SELECT * FROM test_range WHERE period && '[2012-01-03 00:00:00,2012-01-03 12:00:00]';
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(2 rows)

This query could use an index defined like this:

=# CREATE INDEX idx_test_range on test_range USING gist (period);

You can also use these range data types to define exclusion constraints:

CREATE EXTENSION btree_gist ;
CREATE TABLE reservation (room_id int, period tstzrange);
ALTER TABLE reservation ADD EXCLUDE USING GIST (room_id WITH =, period WITH &&);

This means that now it is forbidden to have two records in this table where room_id is equal and period overlaps. The extension btree_gist is required to create a GiST index on room_id (it's an integer, it is usually indexed with a btree index).

=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (2,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');
ERROR: conflicting key value violates exclusion constraint "reservation_room_id_period_excl"
DETAIL: Key (room_id, period)=(1, ("2012-08-23 14:45:00+02","2012-08-23 15:15:00+02"))
conflicts with existing key (room_id, period)=(1, ("2012-08-23 14:00:00+02","2012-08-23 15:00:00+02")).
STATEMENT: INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');

One can also declare new range types.

=Performance improvements=

This version has performance improvements on a very large range of domains (non-exaustive):

* The most visible will probably be the Index Only Scans, which has already been introduced in this document.

* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. 

* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. 

* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption. This is especially useful on virtualized and embedded environments.

* COPY has been improved, it will generate less WAL volume and fewer locks of a table's pages. 

* Statistics are collected on array contents, allowing for better estimations of selectivity on array operations.

* Text-to-anytype concatenation and quote_literal/quote_nullable functions are not volatile any more, enabling better optimization in some cases 

* The system can now track IO durations 

This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.

If you are interested in this timing data, it's better to first check if your system will support it without too much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:

<pre>
$ pg_test_timing
Testing timing overhead for 3 seconds.
Per loop time including overhead: 28.02 nsec
Histogram of timing durations:
< usec: count percent
32: 41 0.00004%
16: 1405 0.00131%
8: 200 0.00019%
4: 388 0.00036%
2: 2982558 2.78523%
1: 104100166 97.21287%
</pre>

Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].

Anyway, here is the data you'll be able to collect if your system is ready for this:

First, you'll get per-database statistics, which will now give accurate informations about which database is doing most I/O:

<pre>
=# SELECT * FROM pg_stat_database WHERE datname = 'mydb';
-[ RECORD 1 ]--+------------------------------
datid | 16384
datname | mydb
numbackends | 1
xact_commit | 270
xact_rollback | 2
blks_read | 1961
blks_hit | 17944
tup_returned | 269035
tup_fetched | 8850
tup_inserted | 16
tup_updated | 4
tup_deleted | 45
conflicts | 0
temp_files | 0
temp_bytes | 0
deadlocks | 0
blk_read_time | 583.774
blk_write_time | 0
stats_reset | 2012-07-03 17:18:54.796817+02
</pre>
We see here that mydb has only consumed 583.774 milliseconds of read time.

Explain will benefit from this too:
<pre>
=# EXPLAIN (analyze,buffers) SELECT count(*) FROM mots ;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
Total runtime: 22.059 ms
</pre>
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).

And last, if you have enabled pg_stat_statements:
<pre>
select * from pg_stat_statements where query ~ 'words';
-[ RECORD 1 ]-------+---------------------------
userid | 10
dbid | 16384
query | select count(*) from words;
calls | 2
total_time | 78.332
rows | 2
shared_blks_hit | 0
shared_blks_read | 986
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 58.427
blk_write_time | 0
</pre>

* As for every version, the optimizer has received its share of improvements 
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it would have been efficient.

This example is straight from the developpers mailing lists :

<pre>
CREATE TABLE a (
a_id serial PRIMARY KEY NOT NULL,
b_id integer
);
CREATE INDEX a__b_id ON a USING btree (b_id);

CREATE TABLE b (
b_id serial NOT NULL,
c_id integer
);
CREATE INDEX b__c_id ON b USING btree (c_id);

CREATE TABLE c (
c_id serial PRIMARY KEY NOT NULL,
value integer UNIQUE
);

INSERT INTO b (b_id, c_id)
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);

INSERT INTO a(b_id)
SELECT g.i FROM generate_series(1, 50000) g(i);

INSERT INTO c(c_id,value)
VALUES (1,1);
</pre>

So we have a referencing b, b referencing c.

Here is an example of a query working badly with PostgreSQL 9.1:

<pre>
EXPLAIN ANALYZE SELECT 1
FROM
c
WHERE
EXISTS (
SELECT *
FROM a
JOIN b USING (b_id)
WHERE b.c_id = c.c_id)
AND c.value = 1;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)
Join Filter: (c.c_id = b.c_id)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)
Index Cond: (value = 1)
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)
Hash Cond: (a.b_id = b.b_id)
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 1954kB
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)
Total runtime: 13.842 ms
</pre>

Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.

Here's what 9.2 does with this query:

<pre>
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)
Index Cond: (value = 1)
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

The «parameterized path» is:
<pre>
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.

This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.

=SP-GiST=

SP-GiST stands for Space Partitionned GiST, GiST being Generalized Search Tree. GiST is an index type, and has been available for quite a while in PostgreSQL. GiST is already very efficient at indexing complex data types, but performance tends to suffer when the source data isn't uniformly distributed. SP-GiST tries to fix that.

As all indexing methods available in PostgreSQL, SP-GiST is a generic indexing method, meaning its purpose is to index whatever you'll throw at it, using operators you'll provide. It means that if you want to create a new datatype, and make it indexable through SP-GiST, you'll have to follow the documented API.

SP-GiST can be used to implement 3 type of indexes: trie (suffix) indexing, Quadtree (data is divided into quadrants), and k-d tree (k-dimensional tree).

For now, SP-GiST is provided with operator families called "quad_point_ops", "kd_point_ops" and "text_ops".

As their names indicate, the first one indexes point types, using a quadtree, the second one indexes point types using a k-d tree, and the third one indexes text, using suffix.

=pg_stat_statements=

This contrib module has received a lot of improvements in this version:

* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.

<pre>
=#SELECT * FROM words WHERE word= 'foo';
word
------
(0 ligne)

=# SELECT * FROM words WHERE word= 'bar';
word
------
bar

=#select * from pg_stat_statements where query like '%words where%';
-[ RECORD 1 ]-------+-----------------------------------
userid | 10
dbid | 16384
query | SELECT * FROM words WHERE word= ?;
calls | 2
total_time | 142.314
rows | 1
shared_blks_hit | 3
shared_blks_read | 5
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 142.165
blk_write_time | 0

</pre>

The two queries are shown as one in pg_stat_statements.

* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That makes it is easier to interpret, and avoids the double-counting there was with PostgreSQL 9.1.

* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.

= Explain improvements=

* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive 

=# EXPLAIN (analyze on,timing off) SELECT * FROM reservation ;
QUERY PLAN
----------------------------------------------------------------------------------------
Seq Scan on reservation (cost=0.00..22.30 rows=1230 width=36) (actual rows=2 loops=1)
Total runtime: 0.045 ms

* Have EXPLAIN ANALYZE report the number of rows rejected by filter steps 

This new feature makes it much easier to know how many rows are removed by a filter (and spot potential places to put indexes):

=# EXPLAIN ANALYZE SELECT * FROM test WHERE a ~ 'tra';
QUERY PLAN
---------------------------------------------------------------------------------------------------------------
Seq Scan on test (cost=0.00..106876.56 rows=2002 width=11) (actual time=2.914..8538.285 rows=120256 loops=1)
Filter: (a ~ 'tra'::text)
Rows Removed by Filter: 5905600
Total runtime: 8549.539 ms
(4 rows)

=Backward compatibility=

These changes may incur regressions in your applications.

==Ensure that xpath() escapes special characters in string values  ==

Before 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
<

'<' Isn't valid XML.
</pre>
With 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
&lt;
</pre>

==Remove hstore's => operator ==
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.

In 9.1:
<pre>
=# SELECT 'a'=>'b';
?column?
----------
"a"=>"b"
(1 row)

=# SELECT pg_typeof('a'=>'b');
pg_typeof
-----------
hstore
(1 row)
</pre>

With 9.2:
<pre>
SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown at character 11
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
STATEMENT: SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown
LINE 1: SELECT 'a'=>'b';
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
</pre>

It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:

<pre>
=# select hstore('a=>b');
hstore
----------
"a"=>"b"
(1 row)

=# select hstore('a','b');
hstore
----------
"a"=>"b"
(1 row)
</pre>
are still two valid ways to input a hstore.

"=>" is removed as an operator as it is a reserved keyword in SQL.

==Have pg_relation_size() and friends return NULL if the object does not exist ==

A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.

==Remove the spclocation field from pg_tablespace ==

The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.

So this field has been removed. To get the tablespace's location, use pg_tablespace_location():

<pre>
=# SELECT *, pg_tablespace_location(oid) AS spclocation FROM pg_tablespace;
spcname | spcowner | spcacl | spcoptions | spclocation
------------+----------+--------+------------+----------------
pg_default | 10 | | |
pg_global | 10 | | |
tmptblspc | 10 | | | /tmp/tmptblspc
</pre>

==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight ==

With PostgreSQL 9.1:

<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341180000
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>

There is no difference in behaviour between a timstamp with or without timezone.

With 9.2:
<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341187200
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.

==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 ==

The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.

Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.

With 9.1:
<pre>
=# SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
1200-07-02
</pre>

With 9.2:
<pre>
SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
2200-07-02
</pre>

==pg_stat_activity and pg_stat_replication's definitions have changed ==

The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:

* current_query disappears and is replaced by two columns:
** state: is the session running a query, waiting
** query: what is the last run (or still running if stat is "active") query
* The column procpid is renamed to pid, to be consistent with other system views

The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.

Here is an example:
<pre>
-[ RECORD 1 ]----+---------------------------------
datid | 16384
datname | postgres
pid | 20804
usesysid | 10
usename | postgres
application_name | psql
client_addr |
client_hostname |
client_port | -1
backend_start | 2012-07-02 15:02:51.146427+02
xact_start | 2012-07-02 15:15:28.386865+02
query_start | 2012-07-02 15:15:30.410834+02
state_change | 2012-07-02 15:15:30.411287+02
waiting | f
state | idle in transaction
query | DELETE FROM test;
</pre>

With PostgreSQL 9.1, all we would have would be «idle in transaction».

As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.
The view pg_stat_replication has also changed. The column procpid is renamed to pid, to also be consistent with other system views.

==Change all SQL-level statistics timing values to float8-stored milliseconds ==

pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.

==postgresql.conf parameters changes ==

* silent_mode has been removed. Use pg_ctl -l postmaster.log
* wal_sender_delay has been removed. It is no longer needed
* custom_variable_classes has been removed. All «classes» are accepted without declaration now
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files

= Other new features =

== DROP INDEX CONCURRENTLY ==

The regular DROP INDEX command takes an exclusive lock on the table. Most of the time, this isn't a problem, because this lock is short-lived. The problem usually occurs when:

* A long-running transaction is running, and has a (shared) lock on the table
* A DROP INDEX is run on this table in another session, asking for an exclusive lock (and waiting for it, as it won't be granted until the long-running transaction ends)

At this point, all other transactions needing to take a shared lock on the table (for a simple SELECT for instance) will have to wait too: their lock acquisition is queued after the DROP INDEX's one.

DROP INDEX CONCURRENTLY works around this and won't lock normal DML statements, just as CREATE INDEX CONCURRENTLY. The limitations are also the same: Since you can only DROP one index with the CONCURRENTLY option, and the CASCADE option is not supported.

== NOT VALID CHECK constraints ==

PostgreSQL 9.1 introduced «NOT VALID» foreign keys. This has been extended to CHECK constraints. Adding a «NOT VALID» constraint on a table means that current data won't be validated, only new and updated rows.

=# CREATE TABLE test (a int);
CREATE TABLE
=# INSERT INTO test SELECT generate_series(1,100);
INSERT 0 100
=# ALTER TABLE test ADD CHECK (a>100) NOT VALID;
ALTER TABLE
=# INSERT INTO test VALUES (99);
ERROR: new row for relation "test" violates check constraint "test_a_check"
DETAIL: Failing row contains (99).
=# INSERT INTO test VALUES (101);
INSERT 0 1

Then, later, we can validate the whole table:

=# ALTER TABLE test VALIDATE CONSTRAINT test_a_check ;
ERROR: check constraint "test_a_check" is violated by some row

Domains, which are types with added constraints, can also be declared as not valid, and validated later.

Check constraints can also be renamed now:

=# ALTER TABLE test RENAME CONSTRAINT test_a_check TO validate_a;
ALTER TABLE

== NO INHERIT constraints ==

Here is another improvement about constraints: they can be declared as not inheritable, which will be useful in partitioned environments. Let's take PostgreSQL documentation example, and see how it improves the situation:

CREATE TABLE measurement (
city_id int not null,
logdate date not null,
peaktemp int,
unitsales int,
CHECK (logdate IS NULL) NO INHERIT
);

CREATE TABLE measurement_y2006m02 (
CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
) INHERITS (measurement);
CREATE TABLE measurement_y2006m03 (
CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' )
) INHERITS (measurement);

INSERT INTO measurement VALUES (1,'2006-02-20',1,1);
ERROR: new row for relation "measurement" violates check constraint "measurement_logdate_check"
DETAIL: Failing row contains (1, 2006-02-20, 1, 1).
INSERT INTO measurement_y2006m02 VALUES (1,'2006-02-20',1,1);
INSERT 0 1

Until now, every check constraint created on measurement would have been inherited by children tables. So adding a constraint forbidding inserts, or allowing only some of them, on the parent table was impossible.

== Reduce ALTER TABLE rewrites ==

A table won't get rewritten anymore during an ALTER TABLE when changing the type of a column in the following cases:

* varchar(x) to varchar(y) when y>=x. It works too if going from varchar(x) to varchar or text (no size limitation)
* numeric(x,z) to numeric(y,z) when y>=x, or to numeric without specifier
* varbit(x) to varbit(y) when y>=x, or to varbit without specifier
* timestamp(x) to timestamp(y) when y>=x or timestamp without specifier
* timestamptz(x) to timestamptz(y) when y>=x or timestamptz without specifier
* interval(x) to interval(y) when y>=x or interval without specifier

== Security barriers and Leakproof ==

This new feature has to do with views security. First, let's explain the problem, with a very simplified example:

=# CREATE TABLE all_data (company_id int, company_data varchar);
CREATE TABLE
=# INSERT INTO all_data VALUES (1,'secret_data_for_company_1');
INSERT 0 1
=# INSERT INTO all_data VALUES (2,'secret_data_for_company_2');
INSERT 0 1
=# CREATE VIEW company1_data AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW

This is a quite classical way of giving access to only a part of a table to a user: we'll create a user for company_id 1, grant to him the right to access company1_data, and deny him the right to access all_data.

The plan to this query is the following:

=# explain SELECT * FROM company1_data ;
QUERY PLAN
----------------------------------------------------------
Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)

Even if there was more data, a sequential scan could still be forced: just "SET enable_indexscan to OFF" and the likes.

So this query reads all the records from all_data, filters them, and returns to the user only the matching rows. There is a way to display scanned records before they are filtered: just create a function with a very low cost, and call it while doing the query:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

This function just has to cost less than the = operator, which costs 1, to be executed first.

The result is this:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

We got access to the record from the second company (in the NOTICE messages).

So this is the first new feature: the view can be declared as implementing "security barriers":

=# CREATE VIEW company1_data WITH (security_barrier) AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

The view is not leaking anymore. The problem, of course, is that there is a performance impact: maybe the "peek" function could have made the query faster, by filtering lots of rows early in the plan. The rows are now filtered in two separate plan steps:

=# explain SELECT * FROM company1_data WHERE peek(company1_data.company_data);
QUERY PLAN
--------------------------------------------------------------------
Subquery Scan on company1_data (cost=0.00..25.44 rows=2 width=36)
Filter: peek((company1_data.company_data)::text)
-> Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)
(4 rows)

This leads to the complementary feature: some function may be declared as "LEAKPROOF", meaning that they won't leak the data they are passed into error or notice messages.

Declaring our peek function as LEAKPROOF is a very bad idea, but let's do it just to demonstrate how it's used:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LEAKPROOF LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

A LEAKPROOF function is executed «normally»:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

Of course, in our case, peek isn't LEAKPROOF and shouldn't be declared as such. Only superuser have the permission to declare a LEAKPROOF function.

== New options for pg_dump ==

Until now, one could ask pg_dump to dump a table's data, or a table's meta-data (DDL statements for creating the table's structure, indexes, constraints). Some meta-data is better restored before the data (the table's structure, check constraints), some is better after the data (indexes, unique constraints, foreign keys…), for performance reasons mostly.

So there are now a few more options:

* --section=pre-data: dump what's needed before restoring the data. Of course, this can be combined with a -t for instance, to specify only one table
* --section=post-data : dump what's needed after restoring the data.
* --section=data: dump the data
* --exclude-table-data: dump everything, except THIS table's data. It means pg_dump will still dump other tables' data.

[[Category:PostgreSQL 9.2]]

Using pg upgrade on Ubuntu/Debian

2012-09-03T10:03:34Z

Intgr: /* What next */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

If you upgraded from 9.0 or earlier and you're using contrib modules or other addons, it is recommended to migrate them to proper extensions. For example:

<pre>CREATE EXTENSION hstore SCHEMA public FROM unpackaged;</pre>

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

Using pg upgrade on Ubuntu/Debian

2012-09-03T09:59:15Z

Intgr:

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== What next ==
Don't forget to merge your configuration changes in postgresql.conf, pg_hba.conf and other files.

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

Using pg upgrade on Ubuntu/Debian

2012-08-29T18:28:12Z

Intgr:

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

When in doubt, use the old but slower pg_upgradecluster method.

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

Using pg upgrade on Ubuntu/Debian

2012-08-29T18:26:55Z

Intgr:

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have only tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade... (For extra performance, you can use the --link option to pg_upgrade; please [http://www.postgresql.org/docs/current/static/pgupgrade.html read pg_upgrade documentation first])

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf -F'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

Using pg upgrade on Ubuntu/Debian

2012-08-29T17:51:18Z

Intgr: /* Upgrading */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have only tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade....

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit
postgres=# \l+
... list of databases ...

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

Using pg upgrade on Ubuntu/Debian

2012-08-29T17:50:35Z

Intgr: /* Prerequisites */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have only tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER, and possibly other modules like postgresql-plpython-VER...

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade....

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

Using pg upgrade on Ubuntu/Debian

2012-08-29T17:42:57Z

Intgr: /* Upgrading */

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have only tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER:

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade....

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

postgres=# select version();
PostgreSQL 9.2rc1 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

Using pg upgrade on Ubuntu/Debian

2012-08-29T17:41:44Z

Intgr: Created page with "Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it…"

Instructions for how to convert/upgrade a PostgreSQL database cluster using pg_upgrade on Ubuntu or Debian. For the sake of example, I'm upgrading from version 9.1 to 9.2, but it should work with any version. Replace any version numbers on the command lines.

== WARNING! ==
'''These instructions are experimental!'''
* This way of upgrading is not yet supported by Ubuntu upstream. Do it at your own risk. I have only tested this on Ubuntu 12.04.
* PostgreSQL 9.2 is still a testing release (currently at RC 1). Do not run it in production!

== Prerequisites ==
First you need to install relevant packages: postgresql-VER and postgresql-server-dev-VER. If you're using contrib extensions, you also need postgresql-contrib-VER:

sudo apt-get install postgresql-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1 \
postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

By default, Ubuntu creates a database cluster named "main" with each installed version. We will use these. Verify that these are configured:

# sudo pg_lsclusters
Version Cluster Port Status Owner Data directory Log file
9.1 main 5432 down postgres /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
9.2 main 5433 down postgres /var/lib/postgresql/9.2/main /var/log/postgresql/postgresql-9.2-main.log

== Upgrading ==
First you need to stop the relevant database clusters. To stop all clusters, run:

sudo /etc/init.d/postgresql stop

(If you don't want to stop all clusters, you can also use pg_ctlcluster to manage them one by one)

Do the upgrade....

cd /tmp
sudo -H -u postgres /usr/lib/postgresql/9.2/bin/pg_upgrade \
-b /usr/lib/postgresql/9.1/bin \
-B /usr/lib/postgresql/9.2/bin \
-d /var/lib/postgresql/9.1/main \
-D /var/lib/postgresql/9.2/main \
-o '-c config_file=/etc/postgresql/9.1/main/postgresql.conf' \
-O '-c config_file=/etc/postgresql/9.2/main/postgresql.conf'

See if it worked...

sudo pg_ctlcluster 9.2 main start
sudo -u postgres psql --cluster 9.2/main

== Troubleshooting ==
==== New cluster database "XXX" is not empty ====
The new cluster version already contains some data. For example, if you want to '''delete all data''' in the 9.2 version cluster, run:

pg_dropcluster 9.2 main
pg_createcluster 9.2 main

==== Where do I get PostgreSQL 9.2? ====
On Ubuntu, you can simply install 9.2 from the semi-official "PPA" repository:

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:pitti/postgresql
sudo apt-get update
sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2

What's new in PostgreSQL 9.2

2012-08-27T16:29:16Z

Intgr: /* Performance improvements */ rm invisible trailing whitespace

{{Languages}}

This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release – PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.

=Major new features=

==Index-only scans ==

In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.

It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.

There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.

The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).

Here is an example.

CREATE TABLE demo_ios (col1 float, col2 float, col3 text);

In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains wont be that big in real life.

INSERT INTO demo_ios SELECT generate_series(1,100000000),random(), 'mynotsolongstring';

SELECT pg_size_pretty(pg_total_relation_size('demo_ios'));
pg_size_pretty
----------------
6512 MB

Let's pretend that the query is this:

SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.01 AND 0.02

In order to use an index only scan on this query, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).

CREATE index idx_demo_ios on demo_ios(col2,col1);

We vacuum the visibility map to be up-to-date:

VACUUM demo_ios;

All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).

Let's first try without Index Only Scans:

SET enable_indexonlyscan to off;

EXPLAIN (analyze,buffers) select col1,col2 FROM demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Rows Removed by Index Recheck: 68098621
Buffers: shared hit=2 read=587779
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Buffers: shared hit=2 read=3835
Total runtime: 364390.127 ms

With Index Only Scans:

explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Heap Fetches: 0
Buffers: shared hit=923073 read=3848
Total runtime: 4297.405 ms

As nothing is free, there are a few things to keep in mind:

* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.
* Gains will probably be smaller in real life situations.

This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.

==Replication improvements ==

Streaming Replication is getting even more polished with this release. One on the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources.

Moreover, in case of a failover, it was very complicated to reconnect all the remaining slaves to the newly promoted master.

To be on the safe side, it was easier to re-synchronize the slaves to the new masters from scratch, meaning that during this failover, only one server was active, and under heavy load, as it was used to rebuild all the slaves.

* With 9.2, a slave can also be a replication master, allowing cascading replication.

Let's build this. We start with an already working 9.2 database.

We set it up for replication:

postgresql.conf:
wal_level=hot_standby #(could be archive too)
max_wal_senders=5
hot_standby=on

You'll probably also want to activate archiving in production, it won't be done here.

pg_hba.conf (do not use trust in production):
host replication replication_user 0.0.0.0/0 md5

Create the user:
create user replication_user replication password 'secret';

Clone the database:

pg_basebackup -h localhost -U replication_user -D data2
Password:

We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf):
port=5433

We add a recovery.conf to tell it how to stream from the master database:
standby_mode = on
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret'

pg_ctl -D data2 start
server starting
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9D0000B8
LOG: database system is ready to accept read only connections

Now, let's add a second slave, which will use this slave:

pg_basebackup -h localhost -U replication_user -D data3 -p 5433
Password:

We edit data3's postgresql.conf to change the port:
port=5434

We modify the recovery.conf to stream from the slave:
standby_mode = on
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'

We start the cluster:
pg_ctl -D data3 start
server starting
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9E000000
LOG: database system is ready to accept read only connections

Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).

* As you may have noticed from the examble, pg_basebackup now works from slaves.

* There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?

pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user

will connect to the master (or a slave), and start creating files:
ls /tmp/new_logs/
00000001000000000000009E.partial

Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.
Remember to rename the last segment to remove the .partial suffix before using it with PITR or other.

* synchronous_commit has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, some people will be interested in this compromise.

==JSON datatype==

The JSON datatype is meant for storing JSON-structured data. It will validate that the input JSON string is correct JSON:

=# SELECT '{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}'::json;
json
-------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
(1 row)

=# SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json at character 8
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...
STATEMENT: SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json
LINE 1: SELECT '{"username","posts":121,"emailaddress":"john@nowhere...
^
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...

You can also convert a row type to JSON:

=#SELECT * FROM demo ;
username | posts | emailaddress
----------+-------+---------------------
john | 121 | john@nowhere.com
mickael | 215 | mickael@nowhere.com
(2 rows)

=# SELECT row_to_json(demo) FROM demo;
row_to_json
-------------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}
(2 rows)

Or an array type:

=# select array_to_json(array_agg(demo)) from demo;
array_to_json
---------------------------------------------------------------------------------------------------------------------------------------------
[{"username":"john","posts":121,"emailaddress":"john@nowhere.com"},{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}]
(1 row)

== Range Types ==

Range types are used to store a range of data of a given type. There are a few pre-defined types. They are integer (int4range), bigint (int8range), numeric (numrange), timestamp without timezone (tsrange), timestamp with timezone (tstzrange), and date (daterange).

Ranges can be made of continuous (numeric, timestamp...) or discrete (integer, date...) data types. They can be open (the bound isn't part of the range) or closed (the bound is part of the range). A bound can also be infinite.

Without these datatypes, most people solve the range problems by using two columns in a table. These range types are much more powerful, as you can use many operators on them:

Here is the intersection between then 1000(open)-2000(closed) and 1000(closed)-1200(closed) numeric range:

SELECT '(1000,2000]'::numrange * '[1000,1200]'::numrange;
?column?
-------------
(1000,1200]
(1 row)

So you can query on things like: «give me all ranges that intersect this»:

=# SELECT * from test_range ;
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-01-02 12:00:00+01"]
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(3 rows)

=# SELECT * FROM test_range WHERE period && '[2012-01-03 00:00:00,2012-01-03 12:00:00]';
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(2 rows)

This query could use an index defined like this:

=# CREATE INDEX idx_test_range on test_range USING gist (period);

You can also use these range data types to define exclusion constraints:

CREATE EXTENSION btree_gist ;
CREATE TABLE reservation (room_id int, period tstzrange);
ALTER TABLE reservation ADD EXCLUDE USING GIST (room_id WITH =, period WITH &&);

This means that now it is forbidden to have two records in this table where room_id is equal and period overlaps. The extension btree_gist is required to create a GiST index on room_id (it's an integer, it is usually indexed with a btree index).

=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (2,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');
ERROR: conflicting key value violates exclusion constraint "reservation_room_id_period_excl"
DETAIL: Key (room_id, period)=(1, ("2012-08-23 14:45:00+02","2012-08-23 15:15:00+02"))
conflicts with existing key (room_id, period)=(1, ("2012-08-23 14:00:00+02","2012-08-23 15:00:00+02")).
STATEMENT: INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');

One can also declare new range types.

=Performance improvements=

This version has performance improvements on a very large range of domains (non-exaustive):

* The most visible will probably be the Index Only Scans, which has already been introduced in this document.

* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. 

* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. 

* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption. This is especially useful on virtualized and embedded environments.

* COPY has been improved, it will generate less WAL volume and less locks of tables's pages. 

* Statistics are collected on array contents, allowing for better estimations of selectivity on array operations.

* Text-to-anytype concatenation and quote_literal/quote_nullable functions are not volatile any more, enabling better optimization in some cases 

* The system can now track IO durations 

This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.

If you are interested in this timing data, it's better to first check if your system will support it without to much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:

<pre>
$ pg_test_timing
Testing timing overhead for 3 seconds.
Per loop time including overhead: 28.02 nsec
Histogram of timing durations:
< usec: count percent
32: 41 0.00004%
16: 1405 0.00131%
8: 200 0.00019%
4: 388 0.00036%
2: 2982558 2.78523%
1: 104100166 97.21287%
</pre>

Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].

Anyway, here is the data you'll be able to collect if your system is ready for this:

First, you'll get per-database statistics, which will now give accurate informations about which database is doing most I/O:

<pre>
=# SELECT * FROM pg_stat_database WHERE datname = 'mydb';
-[ RECORD 1 ]--+------------------------------
datid | 16384
datname | mydb
numbackends | 1
xact_commit | 270
xact_rollback | 2
blks_read | 1961
blks_hit | 17944
tup_returned | 269035
tup_fetched | 8850
tup_inserted | 16
tup_updated | 4
tup_deleted | 45
conflicts | 0
temp_files | 0
temp_bytes | 0
deadlocks | 0
blk_read_time | 583.774
blk_write_time | 0
stats_reset | 2012-07-03 17:18:54.796817+02
</pre>
We see here that mydb has only consumed 583.774 milliseconds of read time.

Explain will benefit from this too:
<pre>
=# EXPLAIN (analyze,buffers) SELECT count(*) FROM mots ;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
Total runtime: 22.059 ms
</pre>
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).

And last, if you have enabled pg_stat_statements:
<pre>
select * from pg_stat_statements where query ~ 'words';
-[ RECORD 1 ]-------+---------------------------
userid | 10
dbid | 16384
query | select count(*) from words;
calls | 2
total_time | 78.332
rows | 2
shared_blks_hit | 0
shared_blks_read | 986
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 58.427
blk_write_time | 0
</pre>

* As for every version, the optimizer has received its share of improvements 
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it would have been efficient.

This example is straight from the developpers mailing lists :

<pre>
CREATE TABLE a (
a_id serial PRIMARY KEY NOT NULL,
b_id integer
);
CREATE INDEX a__b_id ON a USING btree (b_id);

CREATE TABLE b (
b_id serial NOT NULL,
c_id integer
);
CREATE INDEX b__c_id ON b USING btree (c_id);

CREATE TABLE c (
c_id serial PRIMARY KEY NOT NULL,
value integer UNIQUE
);

INSERT INTO b (b_id, c_id)
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);

INSERT INTO a(b_id)
SELECT g.i FROM generate_series(1, 50000) g(i);

INSERT INTO c(c_id,value)
VALUES (1,1);
</pre>

So we have a referencing b, b referencing c.

Here is an example of a query working badly with PostgreSQL 9.1:

<pre>
EXPLAIN ANALYZE SELECT 1
FROM
c
WHERE
EXISTS (
SELECT *
FROM a
JOIN b USING (b_id)
WHERE b.c_id = c.c_id)
AND c.value = 1;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)
Join Filter: (c.c_id = b.c_id)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)
Index Cond: (value = 1)
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)
Hash Cond: (a.b_id = b.b_id)
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 1954kB
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)
Total runtime: 13.842 ms
</pre>

Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.

Here's what 9.2 does with this query:

<pre>
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)
Index Cond: (value = 1)
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

The «parameterized path» is:
<pre>
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.

This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.

=SP-GiST=

SP-GiST stands for Space Partitionned GiST, GiST being Generalized Search Tree. GiST is an index type, and has been available for quite a while in PostgreSQL. GiST is already very efficient at indexing complex data types, but performance tends to suffer when the source data isn't uniformly distributed. SP-GiST tries to fix that.

As all indexing methods available in PostgreSQL, SP-GiST is a generic indexing method, meaning its purpose is to index whatever you'll throw at it, using operators you'll provide. It means that if you want to create a new datatype, and make it indexable through SP-GiST, you'll have to follow the documented API.

SP-GiST can be used to implement 3 type of indexes: trie (suffix) indexing, Quadtree (data is divided into quadrants), and k-d tree (k-dimensional tree).

For now, SP-GiST is provided with operator families called "quad_point_ops", "kd_point_ops" and "text_ops".

As their names indicate, the first one indexes point types, using a quadtree, the second one indexes point types using a k-d tree, and the third one indexes text, using suffix.

=pg_stat_statements=

This contrib module has received a lot of improvements in this version:

* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.

<pre>
=#SELECT * FROM words WHERE word= 'foo';
word
------
(0 ligne)

=# SELECT * FROM words WHERE word= 'bar';
word
------
bar

=#select * from pg_stat_statements where query like '%words where%';
-[ RECORD 1 ]-------+-----------------------------------
userid | 10
dbid | 16384
query | SELECT * FROM words WHERE word= ?;
calls | 2
total_time | 142.314
rows | 1
shared_blks_hit | 3
shared_blks_read | 5
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 142.165
blk_write_time | 0

</pre>

The two queries are shown as one in pg_stat_statements.

* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That way it is easier to use, and avoids the double-counting there was with PostgreSQL 9.1.

* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.

= Explain improvements=

* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive 

=# EXPLAIN (analyze on,timing off) SELECT * FROM reservation ;
QUERY PLAN
----------------------------------------------------------------------------------------
Seq Scan on reservation (cost=0.00..22.30 rows=1230 width=36) (actual rows=2 loops=1)
Total runtime: 0.045 ms

* Have EXPLAIN ANALYZE report the number of rows rejected by filter steps 

This new feature makes it much easier to know how many rows are removed by a filter (and spot potential places to put indexes):

=# EXPLAIN ANALYZE SELECT * FROM test WHERE a ~ 'tra';
QUERY PLAN
---------------------------------------------------------------------------------------------------------------
Seq Scan on test (cost=0.00..106876.56 rows=2002 width=11) (actual time=2.914..8538.285 rows=120256 loops=1)
Filter: (a ~ 'tra'::text)
Rows Removed by Filter: 5905600
Total runtime: 8549.539 ms
(4 rows)

=Backward compatibility=

These changes may incur regressions in your applications.

==Ensure that xpath() escapes special characters in string values  ==

Before 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
<

'<' Isn't valid XML.
</pre>
With 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
&lt;
</pre>

==Remove hstore's => operator ==
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.

In 9.1:
<pre>
=# SELECT 'a'=>'b';
?column?
----------
"a"=>"b"
(1 row)

=# SELECT pg_typeof('a'=>'b');
pg_typeof
-----------
hstore
(1 row)
</pre>

With 9.2:
<pre>
SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown at character 11
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
STATEMENT: SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown
LINE 1: SELECT 'a'=>'b';
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
</pre>

It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:

<pre>
=# select hstore('a=>b');
hstore
----------
"a"=>"b"
(1 row)

=# select hstore('a','b');
hstore
----------
"a"=>"b"
(1 row)
</pre>
are still two valid ways to input a hstore.

"=>" is removed as an operator as it is a reserved keyword in SQL.

==Have pg_relation_size() and friends return NULL if the object does not exist ==

A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.

==Remove the spclocation field from pg_tablespace ==

The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.

So this field has been removed. To get the tablespace's location, use pg_tablespace_location():

<pre>
=# SELECT *, pg_tablespace_location(oid) AS spclocation FROM pg_tablespace;
spcname | spcowner | spcacl | spcoptions | spclocation
------------+----------+--------+------------+----------------
pg_default | 10 | | |
pg_global | 10 | | |
tmptblspc | 10 | | | /tmp/tmptblspc
</pre>

==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight ==

With PostgreSQL 9.1:

<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341180000
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>

There is no difference in behaviour between a timstamp with or without timezone.

With 9.1:
<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341187200
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.

==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 ==

The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.

Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.

With 9.1:
<pre>
=# SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
1200-07-02
</pre>

With 9.2:
<pre>
SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
2200-07-02
</pre>

==pg_stat_activity and pg_stat_replication's definitions have changed ==

The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:

* current_query disappears and is replaced by two columns:
** state: is the session running a query, waiting
** query: what is the last run (or still running if stat is "active") query
* The column procpid is renamed to pid, to be consistent with other system views

The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.

Here is an example:
<pre>
-[ RECORD 1 ]----+---------------------------------
datid | 16384
datname | postgres
pid | 20804
usesysid | 10
usename | postgres
application_name | psql
client_addr |
client_hostname |
client_port | -1
backend_start | 2012-07-02 15:02:51.146427+02
xact_start | 2012-07-02 15:15:28.386865+02
query_start | 2012-07-02 15:15:30.410834+02
state_change | 2012-07-02 15:15:30.411287+02
waiting | f
state | idle in transaction
query | DELETE FROM test;
</pre>

With PostgreSQL 9.1, all we would have would be «idle in transaction».

As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.
The view pg_stat_replication has also changed. The column procpid is renamed to pid, to also be consistent with other system views.

==Change all SQL-level statistics timing values to float8-stored milliseconds ==

pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.

==postgresql.conf parameters changes ==

* silent_mode has been removed. Use pg_ctl -l postmaster.log
* wal_sender_delay has been removed. It is no longer needed
* custom_variable_classes has been removed. All «classes» are accepted without declaration now
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files

= Other new features =

== DROP INDEX CONCURRENTLY ==

The regular DROP INDEX command takes an exclusive lock on the table. Most of the time, this isn't a problem, because this lock is short-lived. The problem usually occurs when:

* A long-running transaction is running, and has a (shared) lock on the table
* A DROP INDEX is run on this table in another session, asking for an exclusive lock (and waiting for it, as it won't be granted until the long-running transaction ends)

At this point, all other transactions needing to take a shared lock on the table (for a simple SELECT for instance) will have to wait too: their lock acquisition is queued after the DROP INDEX's one.

DROP INDEX CONCURRENTLY works around this and won't lock normal DML statements, just as CREATE INDEX CONCURRENTLY. The main limitation is the same: DROP INDEX CONCURRENTLY can't be run in a transaction. Moreover, you can only DROP one index with CONCURRENTLY, and CASCADE isn't supported either.

== NOT VALID CHECK constraints ==

PostgreSQL 9.1 introduced «NOT VALID» foreign keys. This has been extended to CHECK constraints. Adding a «NOT VALID» constraint on a table means that current data won't be validated, only new and updated rows.

=# CREATE TABLE test (a int);
CREATE TABLE
=# INSERT INTO test SELECT generate_series(1,100);
INSERT 0 100
=# ALTER TABLE test ADD CHECK (a>100) NOT VALID;
ALTER TABLE
=# INSERT INTO test VALUES (99);
ERROR: new row for relation "test" violates check constraint "test_a_check"
DETAIL: Failing row contains (99).
=# INSERT INTO test VALUES (101);
INSERT 0 1

Then, later, we can validate the whole table:

=# ALTER TABLE test VALIDATE CONSTRAINT test_a_check ;
ERROR: check constraint "test_a_check" is violated by some row

Domains, which are types with added constraints, can also be declared as not valid, and validated later.

Check constraints can also be renamed now:

=# ALTER TABLE test RENAME CONSTRAINT test_a_check TO validate_a;
ALTER TABLE

Last, but not least, constraints can be declared as not inheritable, which will be useful in partitioned environments. Let's take PostgreSQL documentation example, and see how it improves the situation:

CREATE TABLE measurement (
city_id int not null,
logdate date not null,
peaktemp int,
unitsales int,
CHECK (logdate IS NULL) NO INHERIT
);

CREATE TABLE measurement_y2006m02 (
CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
) INHERITS (measurement);
CREATE TABLE measurement_y2006m03 (
CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' )
) INHERITS (measurement);

INSERT INTO measurement VALUES (1,'2006-02-20',1,1);
ERROR: new row for relation "measurement" violates check constraint "measurement_logdate_check"
DETAIL: Failing row contains (1, 2006-02-20, 1, 1).
INSERT INTO measurement_y2006m02 VALUES (1,'2006-02-20',1,1);
INSERT 0 1

Until now, every check constraint created on measurement would have been inherited by children tables. So adding a constraint forbidding inserts, or allowing only some of them, on the parent table was impossible.

== Reduce ALTER TABLE rewrites ==

A table won't get rewritten anymore during an ALTER TABLE when changing the type of a column in the following cases:

* varchar(x) to varchar(y) when y > x. It works too if going from varchar(x) to varchar or text (no size limitation)
* numeric(x,z) to numeric(y,z) when y>x, or to numeric without specifier
* varbit(x) to varbit(y) when y>x, or to varbit without specifier
* timestamp(x) to timestamp(y) when y>x or timestamp without specifier
* timestamptz(x) to timestamptz(y) when y>x or timestamptz without specifier
* interval(x) to interval(y) when y>x or interval without specifier

== Security barriers and Leakproof ==

This new feature has to do with views security. First, let's explain the problem, with a very simplified example:

=# CREATE TABLE all_data (company_id int, company_data varchar);
CREATE TABLE
# INSERT INTO all_data VALUES (1,'secret_data_for_company_1');
INSERT 0 1
=# INSERT INTO all_data VALUES (2,'secret_data_for_company_2');
INSERT 0 1
=# CREATE VIEW company1_data AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW

This is a quite classical way of giving access to only a part of a table to a user: we'll create a user for company_id 1, grant to him the right to access company1_data, and deny him the right to access all_data.

The plan to this query is the following:

=# explain SELECT * FROM company1_data ;
QUERY PLAN
----------------------------------------------------------
Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)

Even if there was more data, a sequential scan could still be forced: just "SET enable_indexscan to OFF" and the likes.

So this query reads all the records from all_data, filters them, and returns to the user only the matching rows. There is a way to display scanned records before they are filtered: just create a function with a very low cost, and call it while doing the query:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

This function just has to cost less than the = operator, which costs 1, to be executed first.

The result is this:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

We got access to the record from the second company (in the NOTICE messages).

So this is the first new feature: the view can be declared as implementing "security barriers":

=# CREATE VIEW company1_data WITH (security_barrier) AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

The view's not leaking anymore. The problem, of course, is that there is a performance impact: maybe the "peek" function could have made the query faster, by filtering lots of rows early in the plan.

This leads to the complementary feature: some function may be declared as "LEAKPROOF", meaning that they won't leak the data they are passed into error or notice messages.

Declaring our peek function as LEAKPROOF is a very bad idea, but let's do it just to demonstrate how it's used:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LEAKPROOF LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

A LEAKPROOF function is executed «normally»:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

Of course, in our case, peek isn't LEAKPROOF and shouldn't be declared as such. Only superuser have the permission to declare a LEAKPROOF function.

== New options for pg_dump ==

Until now, one could ask pg_dump to dump a table's data, or a table's meta-data (DDL statements for creating the table's structure, indexes, constraints). Some meta-data is better restored before the data (the table's structure, check constraints), some is better after the data (indexes, unique constraints, foreign keys…), for performance reasons mostly.

So there are now a few more options:

* --section=pre-data: dump what's needed before restoring the data. Of course, this can be combined with a -t for instance, to specify one table
* --section=post-data : dump what's needed after restoring the data.
* --section=data: dump the data
* --exclude-table-data: dump everything, except THIS table's data. It means pg_dump will still dump other tables' data.

[[Category:PostgreSQL 9.2]]

What's new in PostgreSQL 9.2

2012-08-27T16:02:48Z

Intgr: /* Performance improvements */ Per 3db6524fe63f0598dcb2b307bb422bc126f2b15d: textanycat, anytextcat, quote_literal, and quote_nullable are now stable

{{Languages}}

This document showcases many of the latest developments in PostgreSQL 9.2, compared to the last major release – PostgreSQL 9.1. There are many improvements in this release, so this wiki page covers many of the more important changes in detail. The full list of changes is itemised in ''Release Notes''.

=Major new features=

==Index-only scans ==

In PostgreSQL, indexes have no "visibility" information. It means that when you access a record by its index, PostgreSQL has to visit the real tuple in the table to be sure it is visible to you: the tuple the index points to may simply be an old version of the record you are looking for.

It can be a very big performance problem: the index is mostly ordered, so accessing its records is quite efficient, while the records may be scattered all over the place (that's a reason why PostgreSQL has a cluster command, but that's another story). In 9.2, PostgreSQL will use an "Index Only Scan" when possible, and not access the record itself if it doesn't need to.

There is still no visibility information in the index. So in order to do this, PostgreSQL uses the visibility map ([http://www.postgresql.org/docs/devel/static/storage-vm.html visibility map]) , which tells it whether the whole content of a (usually) 8K page is visible to all transactions or not. When the index record points to a tuple contained in an «all visible» page, PostgreSQL won't have to access the tuple, it will be able to build it directly from the index. Of course, all the columns requested by the query must be in the index.

The visibility map is maintained by VACUUM (it sets the visible bit), and by the backends doing SQL work (they unset the visible bit).

Here is an example.

CREATE TABLE demo_ios (col1 float, col2 float, col3 text);

In this table, we'll put random data, in order to have "scattered" data. We'll put 100 million records, to have a big recordset, and have it not fit in memory (that's a 4GB-ram machine). This is an ideal case, made for this demo. The gains wont be that big in real life.

INSERT INTO demo_ios SELECT generate_series(1,100000000),random(), 'mynotsolongstring';

SELECT pg_size_pretty(pg_total_relation_size('demo_ios'));
pg_size_pretty
----------------
6512 MB

Let's pretend that the query is this:

SELECT col1,col2 FROM demo_ios where col2 BETWEEN 0.01 AND 0.02

In order to use an index only scan on this query, we need an index on col2,col1 (col2 first, as it is used in the WHERE clause).

CREATE index idx_demo_ios on demo_ios(col2,col1);

We vacuum the visibility map to be up-to-date:

VACUUM demo_ios;

All the timing you'll see below are done on a cold OS and PostgreSQL cache (that's where the gains are, as the purpose on Index Only Scans is to reduce I/O).

Let's first try without Index Only Scans:

SET enable_indexonlyscan to off;

EXPLAIN (analyze,buffers) select col1,col2 FROM demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on demo_ios (cost=25643.01..916484.44 rows=993633 width=16) (actual time=763.391..362963.899 rows=1000392 loops=1)
Recheck Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Rows Removed by Index Recheck: 68098621
Buffers: shared hit=2 read=587779
-> Bitmap Index Scan on idx_demo_ios (cost=0.00..25394.60 rows=993633 width=0) (actual time=759.011..759.011 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Buffers: shared hit=2 read=3835
Total runtime: 364390.127 ms

With Index Only Scans:

explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Heap Fetches: 0
Buffers: shared hit=923073 read=3848
Total runtime: 4297.405 ms

As nothing is free, there are a few things to keep in mind:

* Adding indexes for index only scans obviously adds indexes to your table. So updates will be slower.
* You will index columns that weren't indexed before. So there will be less opportunities for HOT updates.
* Gains will probably be smaller in real life situations.

This required making visibility map changes crash-safe, so visibility map bit changes are now WAL-logged.

==Replication improvements ==

Streaming Replication is getting even more polished with this release. One on the main remaining gripes about streaming replication is that all the slaves have to be connected to the same and unique master, consuming its resources.

Moreover, in case of a failover, it was very complicated to reconnect all the remaining slaves to the newly promoted master.

To be on the safe side, it was easier to re-synchronize the slaves to the new masters from scratch, meaning that during this failover, only one server was active, and under heavy load, as it was used to rebuild all the slaves.

* With 9.2, a slave can also be a replication master, allowing cascading replication.

Let's build this. We start with an already working 9.2 database.

We set it up for replication:

postgresql.conf:
wal_level=hot_standby #(could be archive too)
max_wal_senders=5
hot_standby=on

You'll probably also want to activate archiving in production, it won't be done here.

pg_hba.conf (do not use trust in production):
host replication replication_user 0.0.0.0/0 md5

Create the user:
create user replication_user replication password 'secret';

Clone the database:

pg_basebackup -h localhost -U replication_user -D data2
Password:

We have a brand new cluster in the data2 directory. We'll change the port so that it can start (postgresql.conf):
port=5433

We add a recovery.conf to tell it how to stream from the master database:
standby_mode = on
primary_conninfo = 'host=localhost port=5432 user=replication_user password=secret'

pg_ctl -D data2 start
server starting
LOG: database system was interrupted; last known up at 2012-07-03 17:58:09 CEST
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9D0000B8
LOG: database system is ready to accept read only connections

Now, let's add a second slave, which will use this slave:

pg_basebackup -h localhost -U replication_user -D data3 -p 5433
Password:

We edit data3's postgresql.conf to change the port:
port=5434

We modify the recovery.conf to stream from the slave:
standby_mode = on
primary_conninfo = 'host=localhost port=5433 user=replication_user password=secret' # e.g. 'host=localhost port=5432'

We start the cluster:
pg_ctl -D data3 start
server starting
LOG: database system was interrupted while in recovery at log time 2012-07-03 17:58:09 CEST
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
LOG: creating missing WAL directory "pg_xlog/archive_status"
LOG: entering standby mode
LOG: streaming replication successfully connected to primary
LOG: redo starts at 0/9D000020
LOG: consistent recovery state reached at 0/9E000000
LOG: database system is ready to accept read only connections

Now, everything modified on the master cluster get streamed to the first slave, and from there to the second slave. This second replication has to be monitored from the first slave (the master knows nothing about it).

* As you may have noticed from the examble, pg_basebackup now works from slaves.

* There is another use case that wasn't covered: what if a user didn't care for having a full fledged slave, and only wanted to stream the WAL files to another location, to benefit from the reduced data loss without the burden of maintaining a slave ?

pg_receivexlog is provided just for this purpose: it pretends to be a PostgreSQL slave, but only stores the log files as they are streamed, in a directory:
pg_receivexlog -D /tmp/new_logs -h localhost -U replication_user

will connect to the master (or a slave), and start creating files:
ls /tmp/new_logs/
00000001000000000000009E.partial

Files are of the segment size, so they can be used for a normal recovery of the database. It's the same as an archive command, but with a much smaller granularity.
Remember to rename the last segment to remove the .partial suffix before using it with PITR or other.

* synchronous_commit has a new value: remote_write. It can be used when there is a synchronous slave (synchronous_standby_names is set), meaning that the master doesn't have to wait for the slave to have written the data to disk, only for the slave to have acknowledged the data. With this set, data is protected from a crash on the master, but could still be lost if the slave crashed at the same time (i.e. before having written the in flight data to disk). As this is a quite remote possibility, some people will be interested in this compromise.

==JSON datatype==

The JSON datatype is meant for storing JSON-structured data. It will validate that the input JSON string is correct JSON:

=# SELECT '{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}'::json;
json
-------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
(1 row)

=# SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json at character 8
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...
STATEMENT: SELECT '{"username","posts":121,"emailaddress":"john@nowhere.com"}'::json;
ERROR: invalid input syntax for type json
LINE 1: SELECT '{"username","posts":121,"emailaddress":"john@nowhere...
^
DETAIL: Expected ":", but found ",".
CONTEXT: JSON data, line 1: {"username",...

You can also convert a row type to JSON:

=#SELECT * FROM demo ;
username | posts | emailaddress
----------+-------+---------------------
john | 121 | john@nowhere.com
mickael | 215 | mickael@nowhere.com
(2 rows)

=# SELECT row_to_json(demo) FROM demo;
row_to_json
-------------------------------------------------------------------------
{"username":"john","posts":121,"emailaddress":"john@nowhere.com"}
{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}
(2 rows)

Or an array type:

=# select array_to_json(array_agg(demo)) from demo;
array_to_json
---------------------------------------------------------------------------------------------------------------------------------------------
[{"username":"john","posts":121,"emailaddress":"john@nowhere.com"},{"username":"mickael","posts":215,"emailaddress":"mickael@nowhere.com"}]
(1 row)

== Range Types ==

Range types are used to store a range of data of a given type. There are a few pre-defined types. They are integer (int4range), bigint (int8range), numeric (numrange), timestamp without timezone (tsrange), timestamp with timezone (tstzrange), and date (daterange).

Ranges can be made of continuous (numeric, timestamp...) or discrete (integer, date...) data types. They can be open (the bound isn't part of the range) or closed (the bound is part of the range). A bound can also be infinite.

Without these datatypes, most people solve the range problems by using two columns in a table. These range types are much more powerful, as you can use many operators on them:

Here is the intersection between then 1000(open)-2000(closed) and 1000(closed)-1200(closed) numeric range:

SELECT '(1000,2000]'::numrange * '[1000,1200]'::numrange;
?column?
-------------
(1000,1200]
(1 row)

So you can query on things like: «give me all ranges that intersect this»:

=# SELECT * from test_range ;
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-01-02 12:00:00+01"]
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(3 rows)

=# SELECT * FROM test_range WHERE period && '[2012-01-03 00:00:00,2012-01-03 12:00:00]';
period
-----------------------------------------------------
["2012-01-01 00:00:00+01","2012-03-01 00:00:00+01"]
["2008-01-01 00:00:00+01","2015-01-01 00:00:00+01"]
(2 rows)

This query could use an index defined like this:

=# CREATE INDEX idx_test_range on test_range USING gist (period);

You can also use these range data types to define exclusion constraints:

CREATE EXTENSION btree_gist ;
CREATE TABLE reservation (room_id int, period tstzrange);
ALTER TABLE reservation ADD EXCLUDE USING GIST (room_id WITH =, period WITH &&);

This means that now it is forbidden to have two records in this table where room_id is equal and period overlaps. The extension btree_gist is required to create a GiST index on room_id (it's an integer, it is usually indexed with a btree index).

=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (2,'(2012-08-23 14:00:00,2012-08-23 15:00:00)');
INSERT 0 1
=# INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');
ERROR: conflicting key value violates exclusion constraint "reservation_room_id_period_excl"
DETAIL: Key (room_id, period)=(1, ("2012-08-23 14:45:00+02","2012-08-23 15:15:00+02"))
conflicts with existing key (room_id, period)=(1, ("2012-08-23 14:00:00+02","2012-08-23 15:00:00+02")).
STATEMENT: INSERT INTO reservation VALUES (1,'(2012-08-23 14:45:00,2012-08-23 15:15:00)');

One can also declare new range types.

=Performance improvements=

This version has performance improvements on a very large range of domains (non-exaustive):

* The most visible will probably be the Index Only Scans, which has already been introduced in this document.

* The lock contention of several big locks has been significantly reduced, leading to better multi-processor scalability, for machines with over 32 cores mostly. 

* The performance of in-memory sorts has been improved by up to 25% in some situations, with certain specialized sort functions introduced. 

* An idle PostgreSQL server now makes less wakeups, leading to lower power consumption. This is especially useful on virtualized and embedded environments.

* COPY has been improved, it will generate less WAL volume and less locks of tables's pages. 

* Statistics are collected on array contents, allowing for better estimations of selectivity on array operations.

* Text-to-anytype concatenation and quote_literal/quote_nullable functions are not volatile any more, enabling better optimization in some cases 

* The system can now track IO durations 

This one deserves a little explanation, as it can be a little tricky. Tracking IO durations means asking repeatedly the time to the operating system. Depending on the operating system and the hardware, this can be quite cheap, or extremely costly. The most import factor here is where the system gets its time from. It could be directly retrieved from the processor (TSC), dedicated hardware such as HPET, or an ACPI call. What's most important is that the cost of getting time can vary from a factor of thousands.

If you are interested in this timing data, it's better to first check if your system will support it without to much of a performance hit. PostgreSQL provides you with the pg_test_timing tool:

<pre>
$ pg_test_timing
Testing timing overhead for 3 seconds.
Per loop time including overhead: 28.02 nsec
Histogram of timing durations:
< usec: count percent
32: 41 0.00004%
16: 1405 0.00131%
8: 200 0.00019%
4: 388 0.00036%
2: 2982558 2.78523%
1: 104100166 97.21287%
</pre>

Here, everything is good: getting time costs around 28 nanoseconds, and has a very small variation. Anything under 100 nanoseconds should be good for production. If you get higher values, you may still find a way to tune your system. You'd better check on the [http://www.postgresql.org/docs/9.2/static/pgtesttiming.html documentation].

Anyway, here is the data you'll be able to collect if your system is ready for this:

First, you'll get per-database statistics, which will now give accurate informations about which database is doing most I/O:

<pre>
=# SELECT * FROM pg_stat_database WHERE datname = 'mydb';
-[ RECORD 1 ]--+------------------------------
datid | 16384
datname | mydb
numbackends | 1
xact_commit | 270
xact_rollback | 2
blks_read | 1961
blks_hit | 17944
tup_returned | 269035
tup_fetched | 8850
tup_inserted | 16
tup_updated | 4
tup_deleted | 45
conflicts | 0
temp_files | 0
temp_bytes | 0
deadlocks | 0
blk_read_time | 583.774
blk_write_time | 0
stats_reset | 2012-07-03 17:18:54.796817+02
</pre>
We see here that mydb has only consumed 583.774 milliseconds of read time.

Explain will benefit from this too:
<pre>
=# EXPLAIN (analyze,buffers) SELECT count(*) FROM mots ;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Aggregate (cost=1669.95..1669.96 rows=1 width=0) (actual time=21.943..21.943 rows=1 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
-> Seq Scan on mots (cost=0.00..1434.56 rows=94156 width=0) (actual time=0.059..12.933 rows=94156 loops=1)
Buffers: shared read=493
I/O Timings: read=2.578
Total runtime: 22.059 ms
</pre>
We now have a separate information about the time taken to retrieve data from the operating system. Obviously, here, the data was in the operating system's cache (2 milliseconds to read 493 blocks).

And last, if you have enabled pg_stat_statements:
<pre>
select * from pg_stat_statements where query ~ 'words';
-[ RECORD 1 ]-------+---------------------------
userid | 10
dbid | 16384
query | select count(*) from words;
calls | 2
total_time | 78.332
rows | 2
shared_blks_hit | 0
shared_blks_read | 986
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 58.427
blk_write_time | 0
</pre>

* As for every version, the optimizer has received its share of improvements 
** Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.
** A new feature has been added: parameterized paths. Simply put, it means that a sub-part of a query plan can use parameters it has got from a parent node. It fixes several bad plans that could occur, especially when the optimizer couldn't reorder joins to put nested loops where it would have been efficient.

This example is straight from the developpers mailing lists :

<pre>
CREATE TABLE a (
a_id serial PRIMARY KEY NOT NULL,
b_id integer
);
CREATE INDEX a__b_id ON a USING btree (b_id);

CREATE TABLE b (
b_id serial NOT NULL,
c_id integer
);
CREATE INDEX b__c_id ON b USING btree (c_id);

CREATE TABLE c (
c_id serial PRIMARY KEY NOT NULL,
value integer UNIQUE
);

INSERT INTO b (b_id, c_id)
SELECT g.i, g.i FROM generate_series(1, 50000) g(i);

INSERT INTO a(b_id)
SELECT g.i FROM generate_series(1, 50000) g(i);

INSERT INTO c(c_id,value)
VALUES (1,1);
</pre>

So we have a referencing b, b referencing c.

Here is an example of a query working badly with PostgreSQL 9.1:

<pre>
EXPLAIN ANALYZE SELECT 1
FROM
c
WHERE
EXISTS (
SELECT *
FROM a
JOIN b USING (b_id)
WHERE b.c_id = c.c_id)
AND c.value = 1;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=1347.00..3702.27 rows=1 width=0) (actual time=13.799..13.802 rows=1 loops=1)
Join Filter: (c.c_id = b.c_id)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.006..0.008 rows=1 loops=1)
Index Cond: (value = 1)
-> Hash Join (cost=1347.00..3069.00 rows=50000 width=4) (actual time=13.788..13.788 rows=1 loops=1)
Hash Cond: (a.b_id = b.b_id)
-> Seq Scan on a (cost=0.00..722.00 rows=50000 width=4) (actual time=0.007..0.007 rows=1 loops=1)
-> Hash (cost=722.00..722.00 rows=50000 width=8) (actual time=13.760..13.760 rows=50000 loops=1)
Buckets: 8192 Batches: 1 Memory Usage: 1954kB
-> Seq Scan on b (cost=0.00..722.00 rows=50000 width=8) (actual time=0.008..5.702 rows=50000 loops=1)
Total runtime: 13.842 ms
</pre>

Not that bad, 13 milliseconds. Still, we are doing sequential scans on a and b, when our common sense tells us that c.value=1 should be used to filter rows more aggressively.

Here's what 9.2 does with this query:

<pre>
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
Nested Loop Semi Join (cost=0.00..16.97 rows=1 width=0) (actual time=0.035..0.037 rows=1 loops=1)
-> Index Scan using c_value_key on c (cost=0.00..8.27 rows=1 width=4) (actual time=0.007..0.009 rows=1 loops=1)
Index Cond: (value = 1)
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

The «parameterized path» is:
<pre>
-> Nested Loop (cost=0.00..8.69 rows=1 width=4) (actual time=0.025..0.025 rows=1 loops=1)
-> Index Scan using b__c_id on b (cost=0.00..8.33 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=1)
Index Cond: (c_id = c.c_id)
-> Index Only Scan using a__b_id on a (cost=0.00..0.35 rows=1 width=4) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: (b_id = b.b_id)
Total runtime: 0.089 ms
</pre>

This part of the plan depends on a parent node (c_id=c.c_id). This part of the plan is called each time with a different parameter coming from the parent node.

This plan is of course much faster, as there is no need to fully scan a, and to fully scan AND hash b.

=SP-GiST=

SP-GiST stands for Space Partitionned GiST, GiST being Generalized Search Tree. GiST is an index type, and has been available for quite a while in PostgreSQL. GiST is already very efficient at indexing complex data types, but performance tends to suffer when the source data isn't uniformly distributed. SP-GiST tries to fix that.

As all indexing methods available in PostgreSQL, SP-GiST is a generic indexing method, meaning its purpose is to index whatever you'll throw at it, using operators you'll provide. It means that if you want to create a new datatype, and make it indexable through SP-GiST, you'll have to follow the documented API.

SP-GiST can be used to implement 3 type of indexes: trie (suffix) indexing, Quadtree (data is divided into quadrants), and k-d tree (k-dimensional tree).

For now, SP-GiST is provided with operator families called "quad_point_ops", "kd_point_ops" and "text_ops".

As their names indicate, the first one indexes point types, using a quadtree, the second one indexes point types using a k-d tree, and the third one indexes text, using suffix.

=pg_stat_statements=

This contrib module has received a lot of improvements in this version:

* Queries are normalized: queries that are identical except for their constant values will be considered the same, as long as their post-parse analysis query tree (that is, the internal representation of the query before rule expansion) are the same. This also implies that differences that are not semantically essential to the query, such as variations in whitespace or alias names, or the use of one particular syntax over another equivalent one will not differentiate queries.

<pre>
=#SELECT * FROM words WHERE word= 'foo';
word
------
(0 ligne)

=# SELECT * FROM words WHERE word= 'bar';
word
------
bar

=#select * from pg_stat_statements where query like '%words where%';
-[ RECORD 1 ]-------+-----------------------------------
userid | 10
dbid | 16384
query | SELECT * FROM words WHERE word= ?;
calls | 2
total_time | 142.314
rows | 1
shared_blks_hit | 3
shared_blks_read | 5
shared_blks_dirtied | 0
shared_blks_written | 0
local_blks_hit | 0
local_blks_read | 0
local_blks_dirtied | 0
local_blks_written | 0
temp_blks_read | 0
temp_blks_written | 0
blk_read_time | 142.165
blk_write_time | 0

</pre>

The two queries are shown as one in pg_stat_statements.

* For prepared statements, the execution part (execute statement) is charged on the prepare statement. That way it is easier to use, and avoids the double-counting there was with PostgreSQL 9.1.

* pg_stat_statements displays timing in milliseconds, to be consistent with other system views.

= Explain improvements=

* Timing can now be disabled with EXPLAIN (analyze on, timing off), leading to lower overhead on platforms where getting the current time is expensive 

=# EXPLAIN (analyze on,timing off) SELECT * FROM reservation ;
QUERY PLAN
----------------------------------------------------------------------------------------
Seq Scan on reservation (cost=0.00..22.30 rows=1230 width=36) (actual rows=2 loops=1)
Total runtime: 0.045 ms

* Have EXPLAIN ANALYZE report the number of rows rejected by filter steps 

This new feature makes it much easier to know how many rows are removed by a filter (and spot potential places to put indexes):

=# EXPLAIN ANALYZE SELECT * FROM test WHERE a ~ 'tra';
QUERY PLAN
---------------------------------------------------------------------------------------------------------------
Seq Scan on test (cost=0.00..106876.56 rows=2002 width=11) (actual time=2.914..8538.285 rows=120256 loops=1)
Filter: (a ~ 'tra'::text)
Rows Removed by Filter: 5905600
Total runtime: 8549.539 ms
(4 rows)

=Backward compatibility=

These changes may incur regressions in your applications.

==Ensure that xpath() escapes special characters in string values  ==

Before 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
<

'<' Isn't valid XML.
</pre>
With 9.2:
<pre>
SELECT (XPATH('/*/text()', '<root>&lt;</root>'))[1];
xpath
-------
&lt;
</pre>

==Remove hstore's => operator ==
Up to 9.1, one could use the => operator to create a hstore. Hstore is a contrib, used to store key/values pairs in a column.

In 9.1:
<pre>
=# SELECT 'a'=>'b';
?column?
----------
"a"=>"b"
(1 row)

=# SELECT pg_typeof('a'=>'b');
pg_typeof
-----------
hstore
(1 row)
</pre>

With 9.2:
<pre>
SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown at character 11
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
STATEMENT: SELECT 'a'=>'b';
ERROR: operator does not exist: unknown => unknown
LINE 1: SELECT 'a'=>'b';
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
</pre>

It doesn't mean one cannot use '=>' in hstores, it just isn't an operator anymore:

<pre>
=# select hstore('a=>b');
hstore
----------
"a"=>"b"
(1 row)

=# select hstore('a','b');
hstore
----------
"a"=>"b"
(1 row)
</pre>
are still two valid ways to input a hstore.

"=>" is removed as an operator as it is a reserved keyword in SQL.

==Have pg_relation_size() and friends return NULL if the object does not exist ==

A relation could be dropped by a concurrent session, while one was doing a pg_relation_size on it, leading to a SQL exception. Now, it merely returns NULL for this record.

==Remove the spclocation field from pg_tablespace ==

The spclocation field provided the real location of the tablespace. It was filled in during the CREATE or ALTER TABLESPACE command. So it could be wrong: somebody just had to shutdown the cluster, move the tablespace's directory, re-create the symlink in pg_tblspc, and forget to update the spclocation field. The cluster would still run, as the spclocation wasn't used.

So this field has been removed. To get the tablespace's location, use pg_tablespace_location():

<pre>
=# SELECT *, pg_tablespace_location(oid) AS spclocation FROM pg_tablespace;
spcname | spcowner | spcacl | spcoptions | spclocation
------------+----------+--------+------------+----------------
pg_default | 10 | | |
pg_global | 10 | | |
tmptblspc | 10 | | | /tmp/tmptblspc
</pre>

==Have EXTRACT of a non-timezone-aware value measure the epoch from local midnight, not UTC midnight ==

With PostgreSQL 9.1:

<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341180000
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>

There is no difference in behaviour between a timstamp with or without timezone.

With 9.1:
<pre>
=#SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamp);
date_part
------------
1341187200
(1 row)

=# SELECT extract(epoch FROM '2012-07-02 00:00:00'::timestamptz);
date_part
------------
1341180000
(1 row)
</pre>
When the timestamp has no timezone, the epoch is calculated with the "local midnight", meaning the 1st january of 1970 at midnight, local-time.

==Fix to_date() and to_timestamp() to wrap incomplete dates toward 2020 ==

The wrapping was not consistent between 2 digit dates and 3 digit dates: 2 digit dates always chose the date closest to 2020, 3 digit dates mapped dates from 100 to 999 on 1100 to 1999, and 000 to 099 on 2000 to 2099.

Now PostgreSQL chooses the date closest to 2020, for 2 and 3 digit dates.

With 9.1:
<pre>
=# SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
1200-07-02
</pre>

With 9.2:
<pre>
SELECT to_date('200-07-02','YYY-MM-DD');
to_date
------------
2200-07-02
</pre>

==pg_stat_activity and pg_stat_replication's definitions have changed ==

The view pg_stat_activity has changed. It's not backward compatible, but let's see what this new definition brings us:

* current_query disappears and is replaced by two columns:
** state: is the session running a query, waiting
** query: what is the last run (or still running if stat is "active") query
* The column procpid is renamed to pid, to be consistent with other system views

The benefit is mostly for tracking «idle in transaction» sessions. Up until now, all we could know was that one of these sessions was idle in transaction, meaning it has started a transaction, maybe done some operations, but still not committed. If that session stayed in this state for a while, there was no way of knowing how it got in this state.

Here is an example:
<pre>
-[ RECORD 1 ]----+---------------------------------
datid | 16384
datname | postgres
pid | 20804
usesysid | 10
usename | postgres
application_name | psql
client_addr |
client_hostname |
client_port | -1
backend_start | 2012-07-02 15:02:51.146427+02
xact_start | 2012-07-02 15:15:28.386865+02
query_start | 2012-07-02 15:15:30.410834+02
state_change | 2012-07-02 15:15:30.411287+02
waiting | f
state | idle in transaction
query | DELETE FROM test;
</pre>

With PostgreSQL 9.1, all we would have would be «idle in transaction».

As this change was backward-incompatible, procpid was also renamed to pid, to be more consistent with other system views.
The view pg_stat_replication has also changed. The column procpid is renamed to pid, to also be consistent with other system views.

==Change all SQL-level statistics timing values to float8-stored milliseconds ==

pg_stat_user_functions.total_time, pg_stat_user_functions.self_time, pg_stat_xact_user_functions.total_time, pg_stat_xact_user_functions.self_time, and pg_stat_statements.total_time (contrib) are now in milliseconds, to be consistent with the rest of the timing values.

==postgresql.conf parameters changes ==

* silent_mode has been removed. Use pg_ctl -l postmaster.log
* wal_sender_delay has been removed. It is no longer needed
* custom_variable_classes has been removed. All «classes» are accepted without declaration now
* ssl_ca_file, ssl_cert_file, ssl_crl_file, ssl_key_file have been added, meaning you can now specify the ssl files

= Other new features =

== DROP INDEX CONCURRENTLY ==

The regular DROP INDEX command takes an exclusive lock on the table. Most of the time, this isn't a problem, because this lock is short-lived. The problem usually occurs when:

* A long-running transaction is running, and has a (shared) lock on the table
* A DROP INDEX is run on this table in another session, asking for an exclusive lock (and waiting for it, as it won't be granted until the long-running transaction ends)

At this point, all other transactions needing to take a shared lock on the table (for a simple SELECT for instance) will have to wait too: their lock acquisition is queued after the DROP INDEX's one.

DROP INDEX CONCURRENTLY works around this and won't lock normal DML statements, just as CREATE INDEX CONCURRENTLY. The main limitation is the same: DROP INDEX CONCURRENTLY can't be run in a transaction. Moreover, you can only DROP one index with CONCURRENTLY, and CASCADE isn't supported either.

== NOT VALID CHECK constraints ==

PostgreSQL 9.1 introduced «NOT VALID» foreign keys. This has been extended to CHECK constraints. Adding a «NOT VALID» constraint on a table means that current data won't be validated, only new and updated rows.

=# CREATE TABLE test (a int);
CREATE TABLE
=# INSERT INTO test SELECT generate_series(1,100);
INSERT 0 100
=# ALTER TABLE test ADD CHECK (a>100) NOT VALID;
ALTER TABLE
=# INSERT INTO test VALUES (99);
ERROR: new row for relation "test" violates check constraint "test_a_check"
DETAIL: Failing row contains (99).
=# INSERT INTO test VALUES (101);
INSERT 0 1

Then, later, we can validate the whole table:

=# ALTER TABLE test VALIDATE CONSTRAINT test_a_check ;
ERROR: check constraint "test_a_check" is violated by some row

Domains, which are types with added constraints, can also be declared as not valid, and validated later.

Check constraints can also be renamed now:

=# ALTER TABLE test RENAME CONSTRAINT test_a_check TO validate_a;
ALTER TABLE

Last, but not least, constraints can be declared as not inheritable, which will be useful in partitioned environments. Let's take PostgreSQL documentation example, and see how it improves the situation:

CREATE TABLE measurement (
city_id int not null,
logdate date not null,
peaktemp int,
unitsales int,
CHECK (logdate IS NULL) NO INHERIT
);

CREATE TABLE measurement_y2006m02 (
CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
) INHERITS (measurement);
CREATE TABLE measurement_y2006m03 (
CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' )
) INHERITS (measurement);

INSERT INTO measurement VALUES (1,'2006-02-20',1,1);
ERROR: new row for relation "measurement" violates check constraint "measurement_logdate_check"
DETAIL: Failing row contains (1, 2006-02-20, 1, 1).
INSERT INTO measurement_y2006m02 VALUES (1,'2006-02-20',1,1);
INSERT 0 1

Until now, every check constraint created on measurement would have been inherited by children tables. So adding a constraint forbidding inserts, or allowing only some of them, on the parent table was impossible.

== Reduce ALTER TABLE rewrites ==

A table won't get rewritten anymore during an ALTER TABLE when changing the type of a column in the following cases:

* varchar(x) to varchar(y) when y > x. It works too if going from varchar(x) to varchar or text (no size limitation)
* numeric(x,z) to numeric(y,z) when y>x, or to numeric without specifier
* varbit(x) to varbit(y) when y>x, or to varbit without specifier
* timestamp(x) to timestamp(y) when y>x or timestamp without specifier
* timestamptz(x) to timestamptz(y) when y>x or timestamptz without specifier
* interval(x) to interval(y) when y>x or interval without specifier

== Security barriers and Leakproof ==

This new feature has to do with views security. First, let's explain the problem, with a very simplified example:

=# CREATE TABLE all_data (company_id int, company_data varchar);
CREATE TABLE
# INSERT INTO all_data VALUES (1,'secret_data_for_company_1');
INSERT 0 1
=# INSERT INTO all_data VALUES (2,'secret_data_for_company_2');
INSERT 0 1
=# CREATE VIEW company1_data AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW

This is a quite classical way of giving access to only a part of a table to a user: we'll create a user for company_id 1, grant to him the right to access company1_data, and deny him the right to access all_data.

The plan to this query is the following:

=# explain SELECT * FROM company1_data ;
QUERY PLAN
----------------------------------------------------------
Seq Scan on all_data (cost=0.00..25.38 rows=6 width=36)
Filter: (company_id = 1)

Even if there was more data, a sequential scan could still be forced: just "SET enable_indexscan to OFF" and the likes.

So this query reads all the records from all_data, filters them, and returns to the user only the matching rows. There is a way to display scanned records before they are filtered: just create a function with a very low cost, and call it while doing the query:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

This function just has to cost less than the = operator, which costs 1, to be executed first.

The result is this:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

We got access to the record from the second company (in the NOTICE messages).

So this is the first new feature: the view can be declared as implementing "security barriers":

=# CREATE VIEW company1_data WITH (security_barrier) AS SELECT * FROM all_data WHERE company_id = 1;
CREATE VIEW
=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

The view's not leaking anymore. The problem, of course, is that there is a performance impact: maybe the "peek" function could have made the query faster, by filtering lots of rows early in the plan.

This leads to the complementary feature: some function may be declared as "LEAKPROOF", meaning that they won't leak the data they are passed into error or notice messages.

Declaring our peek function as LEAKPROOF is a very bad idea, but let's do it just to demonstrate how it's used:

CREATE OR REPLACE FUNCTION peek(text) RETURNS boolean LEAKPROOF LANGUAGE plpgsql AS
$$
BEGIN
RAISE NOTICE '%',$1;
RETURN true;
END
$$
COST 0.1;

A LEAKPROOF function is executed «normally»:

=# SELECT * FROM company1_data WHERE peek(company1_data.company_data);
NOTICE: secret_data_for_company_1
NOTICE: secret_data_for_company_2
company_id | company_data
------------+---------------------------
1 | secret_data_for_company_1
(1 row)

Of course, in our case, peek isn't LEAKPROOF and shouldn't be declared as such. Only superuser have the permission to declare a LEAKPROOF function.

== New options for pg_dump ==

Until now, one could ask pg_dump to dump a table's data, or a table's meta-data (DDL statements for creating the table's structure, indexes, constraints). Some meta-data is better restored before the data (the table's structure, check constraints), some is better after the data (indexes, unique constraints, foreign keys…), for performance reasons mostly.

So there are now a few more options:

* --section=pre-data: dump what's needed before restoring the data. Of course, this can be combined with a -t for instance, to specify one table
* --section=post-data : dump what's needed after restoring the data.
* --section=data: dump the data
* --exclude-table-data: dump everything, except THIS table's data. It means pg_dump will still dump other tables' data.

[[Category:PostgreSQL 9.2]]