PostgreSQL wiki - User contributions [en]

PgCon 2013 Developer Meeting

Kgrittn — Mon, 13 May 2013 20:18:10 GMT

Kgrittn: /* Proposed Agenda Items */ Add name to proposed agenda item

A meeting of the most active PostgreSQL developers is being planned for Wednesday 22nd May, 2013 near the University of Ottawa, prior to pgCon 2013. In order to keep the numbers manageable, this meeting is '''by invitation only'''. Unfortunately it is quite possible that we've overlooked important code developers during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that this year the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.3 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies, unlike in previous years.

This is a PostgreSQL Community event. Room and refreshments/food sponsored by EnterpriseDB. Other companies sponsored attendance for their developers.

== Time & Location ==

The meeting will be from 8:30AM to 5PM, and will be in the "Red Experience" room at:

Novotel Ottawa
33 Nicholas Street
Ottawa
Ontario
K1N 9M7

Food and drink will be provided throughout the day, including breakfast from 8AM.

[http://maps.google.ca/maps?f=q&source=s_q&hl=en&geocode=&q=novotel+ottawa&aq=&sll=49.891235,-97.15369&sspn=36.237851,79.013672&ie=UTF8&hq=novotel+ottawa&hnear=&ll=45.421528,-75.683699&spn=0.036869,0.077162&z=14&iwloc=A&layer=c&cbll=45.425741,-75.689638&panoid=Z4FUGnkZkdHAOkIxyjjS9Q&cbp=12,25.83,,0,-0.6 View on Google Maps]

== Attendees ==

The following people have RSVPed to the meeting (in alphabetical order, by surname):

* Josh Berkus (secretary)
* Jeff Davis
* Andrew Dunstan
* Peter Eisentraut
* Dimitri Fontaine
* Andres Freund
* Stephen Frost
* Peter Geoghegan
* Kevin Grittner
* Robert Haas
* Magnus Hagander
* KaiGai Kohei
* Alexander Korotkov
* Tom Lane
* Fujii Masao
* Noah Misch
* Bruce Momjian
* Dave Page (chair)
* Simon Riggs

== Proposed Agenda Items ==

Please list proposed agenda items here:

* 9.4 Commitfest schedule
* [http://wiki.postgresql.org/wiki/Parallel_Query_Execution Parallel Query Execution] (Bruce, Noah)
* logical changeset generation review & integration (Andres)
* utilization of upcoming non-volatile RAM device (Kaigai)
* pluggable plan/exec nodes (Kaigai)
** to offload targetlist calculation, sorting, aggregates, ...
* [[GIN generalization]] (Alexander)
* An Extensibility Roadmap (dim)
* Representing severity - derive severity from SQLSTATE (Peter Geoghegan - see http://www.postgresql.org/message-id/CA+TgmoZEjq7va+SfDZQwk6E4emEWThENNyxfqEGhB3iuoT1OJw@mail.gmail.com)
* Error logging infrastructure - store normalized statistics about errors in a circular buffer (Peter Geoghegan). Arguably this could be discussed alongside SQLSTATE item.
* Failback with backup (Fujii Masao - related discussion is: http://www.postgresql.org/message-id/CAF8Q-Gxg3PQTf71NVECe-6OzRaew5pWhk7yQtbJgWrFu513s+Q@mail.gmail.com)
* Volume Management (Stephen Frost - wiki page will be forthcoming before the meeting)
* AXLE Project - Big data analytics for Postgres (Simon Riggs) - an overview of the feature plan, how project works and what community can expect
* Incremental maintenance of materialized views (Kevin) - differential REFRESH and infrastructure for ''counting'' algorithm

== Agenda ==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter
|- style="font-style:italic;background-color:lightgray;"
|08:00
|Breakfast
|

|- style="font-style:italic;background-color:lightgray;"
|08:30 - 08:45
|Welcome and introductions
|Dave Page

|-
|08:45 - 09:45
|Parallel Query Execution
|Bruce/Noah

|-
|09:45 - 10:30
|Logical changeset generation review & integration
|Andres

|- style="font-style:italic;background-color:lightgray;"
|10:30 - 10:45
|Coffee break
|

|-
|10:45 - 11:00
|Utilization of upcoming non-volatile RAM devices
|KaiGai

|-
|11:00 - 11:30
|Pluggable plan/exec nodes
|KaiGai

|-
|11:30 - 11:50
|Representing severity
|Peter G.

|-
|11:50 - 12:30
|Error logging infrastructure
|Peter G.

|- style="font-style:italic;background-color:lightgray;"
|12:30 - 13:30
|Lunch
|

|-
|13:30 - 14:15
|GIN generalization
|Alexander

|-
|14:15 - 15:00
|An Extensibility Roadmap
|Dimitri

|- style="font-style:italic;background-color:lightgray;"
|15:00 - 15:15
|Tea break
|

|-
|15:15 - 15:30
|9.4 Commitfest schedule
|All

|-
|15:30 - 16:45
|Goals, priorities, and resources for 9.4
|All

|- style="font-style:italic;background-color:lightgray;"
|16:45 - 17:00
|Any other business/group photo
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|17:00
|Finish
|
|}

PgCon 2013 Developer Meeting

Kgrittn — Mon, 13 May 2013 20:17:03 GMT

Kgrittn: /* Proposed Agenda Items */ Add proposed agenda item for incremental maintenance of materialized views

A meeting of the most active PostgreSQL developers is being planned for Wednesday 22nd May, 2013 near the University of Ottawa, prior to pgCon 2013. In order to keep the numbers manageable, this meeting is '''by invitation only'''. Unfortunately it is quite possible that we've overlooked important code developers during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that this year the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.3 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies, unlike in previous years.

This is a PostgreSQL Community event. Room and refreshments/food sponsored by EnterpriseDB. Other companies sponsored attendance for their developers.

== Time & Location ==

The meeting will be from 8:30AM to 5PM, and will be in the "Red Experience" room at:

Novotel Ottawa
33 Nicholas Street
Ottawa
Ontario
K1N 9M7

Food and drink will be provided throughout the day, including breakfast from 8AM.

[http://maps.google.ca/maps?f=q&source=s_q&hl=en&geocode=&q=novotel+ottawa&aq=&sll=49.891235,-97.15369&sspn=36.237851,79.013672&ie=UTF8&hq=novotel+ottawa&hnear=&ll=45.421528,-75.683699&spn=0.036869,0.077162&z=14&iwloc=A&layer=c&cbll=45.425741,-75.689638&panoid=Z4FUGnkZkdHAOkIxyjjS9Q&cbp=12,25.83,,0,-0.6 View on Google Maps]

== Attendees ==

The following people have RSVPed to the meeting (in alphabetical order, by surname):

* Josh Berkus (secretary)
* Jeff Davis
* Andrew Dunstan
* Peter Eisentraut
* Dimitri Fontaine
* Andres Freund
* Stephen Frost
* Peter Geoghegan
* Kevin Grittner
* Robert Haas
* Magnus Hagander
* KaiGai Kohei
* Alexander Korotkov
* Tom Lane
* Fujii Masao
* Noah Misch
* Bruce Momjian
* Dave Page (chair)
* Simon Riggs

== Proposed Agenda Items ==

Please list proposed agenda items here:

* 9.4 Commitfest schedule
* [http://wiki.postgresql.org/wiki/Parallel_Query_Execution Parallel Query Execution] (Bruce, Noah)
* logical changeset generation review & integration (Andres)
* utilization of upcoming non-volatile RAM device (Kaigai)
* pluggable plan/exec nodes (Kaigai)
** to offload targetlist calculation, sorting, aggregates, ...
* [[GIN generalization]] (Alexander)
* An Extensibility Roadmap (dim)
* Representing severity - derive severity from SQLSTATE (Peter Geoghegan - see http://www.postgresql.org/message-id/CA+TgmoZEjq7va+SfDZQwk6E4emEWThENNyxfqEGhB3iuoT1OJw@mail.gmail.com)
* Error logging infrastructure - store normalized statistics about errors in a circular buffer (Peter Geoghegan). Arguably this could be discussed alongside SQLSTATE item.
* Failback with backup (Fujii Masao - related discussion is: http://www.postgresql.org/message-id/CAF8Q-Gxg3PQTf71NVECe-6OzRaew5pWhk7yQtbJgWrFu513s+Q@mail.gmail.com)
* Volume Management (Stephen Frost - wiki page will be forthcoming before the meeting)
* AXLE Project - Big data analytics for Postgres (Simon Riggs) - an overview of the feature plan, how project works and what community can expect
* Incremental maintenance of materialized views - differential REFRESH and infrastructure for ''counting'' algorithm

== Agenda ==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter
|- style="font-style:italic;background-color:lightgray;"
|08:00
|Breakfast
|

|- style="font-style:italic;background-color:lightgray;"
|08:30 - 08:45
|Welcome and introductions
|Dave Page

|-
|08:45 - 09:45
|Parallel Query Execution
|Bruce/Noah

|-
|09:45 - 10:30
|Logical changeset generation review & integration
|Andres

|- style="font-style:italic;background-color:lightgray;"
|10:30 - 10:45
|Coffee break
|

|-
|10:45 - 11:00
|Utilization of upcoming non-volatile RAM devices
|KaiGai

|-
|11:00 - 11:30
|Pluggable plan/exec nodes
|KaiGai

|-
|11:30 - 11:50
|Representing severity
|Peter G.

|-
|11:50 - 12:30
|Error logging infrastructure
|Peter G.

|- style="font-style:italic;background-color:lightgray;"
|12:30 - 13:30
|Lunch
|

|-
|13:30 - 14:15
|GIN generalization
|Alexander

|-
|14:15 - 15:00
|An Extensibility Roadmap
|Dimitri

|- style="font-style:italic;background-color:lightgray;"
|15:00 - 15:15
|Tea break
|

|-
|15:15 - 15:30
|9.4 Commitfest schedule
|All

|-
|15:30 - 16:45
|Goals, priorities, and resources for 9.4
|All

|- style="font-style:italic;background-color:lightgray;"
|16:45 - 17:00
|Any other business/group photo
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|17:00
|Finish
|
|}

What's new in PostgreSQL 9.3

Kgrittn — Wed, 08 May 2013 20:56:34 GMT

Kgrittn: /* Materialized Views */ Add a link to the overview in the docs (which is located under "The Rule System")

This page contains additional information about PostgreSQL Version 9.3's features, including descriptions, testing information, and usage information. See also the [http://www.postgresql.org/docs/devel/static/release-9-3.html Release Notes] and [[PostgreSQL_9.3_Blog_Posts|this page]] for a list of blog posts explaining some of the new features.

= New features =

''... in no particular order ...''

Items marked as "(DONE)" have a basic description + links, but can of course be improved / expanded.

* Writeable Foreign Tables: write to external databases as well as read from them
* pgsql_fdw driver for federation of PostgreSQL databases
* VIEW features
** Automatically updatable VIEWs (DONE)
** MATERIALIZED VIEWs declaration (DONE)
** Recursive view declaration (DONE)
* LATERAL JOINs
* Additional JSON constructor and extractor functions
* Switch to Posix shared memory and mmap(). (DONE)
* Indexed regular expression search
* Disk page checksums to detect filesystem failures
* Replication improvements
** Streaming-only remastering of replicas
** Streaming replication protocol is now architecture-independent.
** Faster promotion of a streaming standby to primary ("Standby promotion is almost instant, allowing 99.999% availability for a replicated cluster.")
* Performance and locking improvements for Foreign Key locks
* Parallel pg_dump for faster backups (DONE)
* Directories for configuration files (DONE)
* pg_isready database connection checker (DONE)
* 64-bit Large Object API
* COPY FREEZE for reduced IO bulk loading
* User-defined background workers for automating database tasks (DONE)

== Configuration directive 'include_dir' ==

In addition to including separate configuration files via the 'include' directive, postgresql.conf now also provides the 'include_dir' directive which reads all files ending in ".conf" in the specified directory or directories.

Directories can be specified either as an absolute path or relative from the location of the main configuration file. Directories will be read in the order they occur, while files will be read sorted by C locale rules. It is possible for included files to contain their own 'include_dir' directives.

'''Links'''

* [http://www.postgresql.org/docs/devel/static/config-setting.html#CONFIG-INCLUDES Documentation]

== Custom Background Workers ==

This functionality enables modules to register themselves as "background worker processes", effectively operating as customised server processes. This is a powerful new feature with a wide variety of possible use cases, such as monitoring server activity, performing tasks at pre-defined intervals, customised logging etc.

Background worker processes can attach to PostgreSQL's shared memory area and to connect to databases internally; by linking to libpq they can also connect to the server in the same way as a regular client application. Background worker processes are written in C, and as server processes they have unrestricted access to all data and can potentially impact other server processes, meaning so they represent a potential security / stability risk. Consequently background worker processes should be developed and deployed with appropriate caution.

Providing an example would go beyond the scope of this article; please refer to the blogs linked below, which provide annotated sample code. The PostgreSQL source also contains a sample background worker process in contrib/worker_spi.

'''Links'''

* [http://www.postgresql.org/docs/devel/static/bgworker.html Documentation]
* [http://www.depesz.com/2012/12/07/waiting-for-9-3-background-worker-processes/ Background worker processes]
* [http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-handling-signals-with-custom-bgworkers/ Postgres 9.3 feature highlight: handling signals with custom bgworkers]
* [http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-custom-background-workers/ Custom background workers]
* [http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-hello-world-with-custom-bgworkers/ "Hello World" with custom bgworkers]

== Parallel pg_dump for faster backups ==

The new ''-j '''njobs''''' (''--jobs=''''njobs'''''') option enables pg_dump to dump '''njobs''' tables simultaneously, reducing the time it takes to dump a database. Example:

pg_dump -U postgres -j4 -Fd -f /tmp/mydb-dump mydb

This dumps the contents of database "mydb" to the directory "/tmp/mydb-dump" using four simultaneous connections.

Caveats:
* Parallel dumps can only be in directory format
* Parellel dumps will place more load on the database, although total dump time should be shorter
* pg_dump will open njobs + 1 connections to the database, so max_connections should be set appropriately
* Requesting exclusive locks on database objects while running a parallel dump could cause the dump to fail
* Parallel dumps from pre-9.2 servers need special attention

An ad-hoc test of this feature on a 4.5GB database (which compresses to around 370MB as a dump) with different values of ''-j '' produced following timings:

* (''no -j''): 1m3s
* -j2: 0m28s
* -j3: 0m24s
* -j4: 0m24s
* -j5: 0m25s

'''Links'''

* [http://www.postgresql.org/docs/devel/static/app-pgdump.html pg_dump documentation]
* [http://www.depesz.com/2013/03/26/2646/ Waiting for 9.3 – Add parallel pg_dump option]
* [http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-parallel-pg_dump/ Postgres 9.3 feature highlight: parallel pg_dump]

== 'pg_isready' server monitoring tool ==

pg_isready is a wrapper for PQping created as a standard client application. It accepts a libpq-style connection string and returns one of four exit statuses:

* 0: server is accepting connections normally
* 1: server is rejecting connections (for example during startup)
* 2: server did not response to the connection attempt
* 3: no connection attempt was made (e.g. due to invalid connection parameters)

Example usage:

barwick@localhost:~$ pg_isready
/tmp:5432 - accepting connections
barwick@localhost:~$ pg_isready --quiet && echo "OK"
OK
barwick@localhost:~$ pg_isready -p5431 -h localhost
localhost:5431 - accepting connections
barwick@localhost:~$ pg_isready -h example.com
example.com:5432 - no response

'''Links'''

* [http://www.postgresql.org/docs/devel/static/app-pg-isready.html Documentation]
* [http://www.depesz.com/2013/01/26/waiting-for-9-3-pg_isready/ pg_isready]
* [http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-server-monitoring-with-pg_isready/ Server monitoring with pg_isready]

== pgsql_fdw driver for federation of PostgreSQL databases ==

New PostgreSQL-to-PostgreSQL foreign data wrapper, which allows writes and "pushdown" of some query clauses to the external server.

== Switch to Posix shared memory and mmap() ==

In 9.3, PostgreSQL has switched from using SysV shared memory to using Posix shared memory and mmap for memory management. This allows easier installation and configuration of PostgreSQL, and means that except in usual cases, system parameters such as SHMMAX and SHMALL no longer need to be adjusted. We need users to rigorously test and ensure that no memory management issues have been introduced by the change.

'''Links'''

* [http://www.postgresql.org/docs/devel/static/kernel-resources.html#SYSVIPC Documentation]

== VIEW Features ==
=== Materialized Views ===

Materialized views are a special kind of view which cache the view's output as a physical table, rather than executing the underlying query on every access. Conceptually they are similar to "CREATE TABLE AS", but store the view definition so it can be easily refreshed.

Note that materialized views cannot be auto-refreshed; refreshes are not incremental; and the base table cannot be manipulated. It is currently uncertain whether they will be automatically populated by pg_restore (''??? need to confirm this'')

'''Contrived example'''

Create and populate a table with some arbitrary data:

CREATE TABLE matview_test_table (
id SERIAL PRIMARY KEY,
ts TIMESTAMPTZ NOT NULL
)

INSERT INTO matview_test_table VALUES (
DEFAULT,
((NOW() - '2 days'::INTERVAL) + (generate_series(1,1000) || ' seconds')::INTERVAL)::TIMESTAMPTZ
)

Create a materialized view which lists the 5 most recent entries:

CREATE MATERIALIZED VIEW matview_test_view AS
SELECT id, ts
FROM matview_test_table
ORDER BY id DESC
LIMIT 5

postgres=# SELECT * from matview_test_view ;
id | ts
------+-------------------------------
1000 | 2013-05-06 12:02:10.974711+09
999 | 2013-05-06 12:02:09.974711+09
998 | 2013-05-06 12:02:08.974711+09
997 | 2013-05-06 12:02:07.974711+09
996 | 2013-05-06 12:02:06.974711+09
(5 rows)

Add more data to the table:

INSERT INTO matview_test_table VALUES (
DEFAULT,
((NOW() - '1 days'::INTERVAL) + (generate_series(1,1000) || ' seconds')::INTERVAL)::TIMESTAMPTZ
)

View output does not change:

postgres=# SELECT * from matview_test_view ;
id | ts
------+-------------------------------
1000 | 2013-05-06 12:02:10.974711+09
999 | 2013-05-06 12:02:09.974711+09
998 | 2013-05-06 12:02:08.974711+09
997 | 2013-05-06 12:02:07.974711+09
996 | 2013-05-06 12:02:06.974711+09
(5 rows)

Refresh the view to display the latest table entries:

postgres=# REFRESH MATERIALIZED VIEW matview_test_view ;
REFRESH MATERIALIZED VIEW
postgres=# SELECT * from matview_test_view ;
id | ts
------+-------------------------------
2001 | 2013-05-07 12:03:10.696626+09
2000 | 2013-05-07 12:03:09.696626+09
1999 | 2013-05-07 12:03:08.696626+09
1998 | 2013-05-07 12:03:07.696626+09
1997 | 2013-05-07 12:03:06.696626+09
(5 rows)

The links below contain more detailed information and examples.

'''Links'''
* Documentation:
** [http://www.postgresql.org/docs/devel/static/rules-materializedviews.html Overview]
** [http://www.postgresql.org/docs/devel/static/sql-creatematerializedview.html CREATE command]
* [http://www.depesz.com/2013/03/04/waiting-for-9-3-add-a-materialized-view-relations/ Waiting for 9.3 – Add a materialized view relations]
* [http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-materialized-views/ Postgres 9.3 feature highlight: Materialized views]

=== Recursive View Syntax ===

The CREATE RECURSIVE VIEW syntax provides a shorthand way of formulating a recursive common table expression (CTE) as a view.

Taking the example from the [http://www.postgresql.org/docs/current/static/queries-with.html#QUERIES-WITH-SELECT CTE documentation]:

WITH RECURSIVE t(n) AS (
VALUES (1)
UNION ALL
SELECT n+1 FROM t WHERE n < 100
)
SELECT * FROM t;

This can be created as a recursive view as follows:

CREATE RECURSIVE VIEW t(n) AS
VALUES (1)
UNION ALL
SELECT n+1 FROM t WHERE n < 100;

'''Links'''
* [http://www.postgresql.org/docs/devel/static/sql-createview.html Documentation]
* [http://www.depesz.com/2013/03/04/waiting-for-9-3-add-create-recursive-view-syntax/ Waiting for 9.3 – Add CREATE RECURSIVE VIEW syntax]

=== Updatable Views ===

Simple views can now be updated in the same way as regular tables. The view can only reference one table (or another updatable view) and must not contain more complex operators, join types etc.

If the view has a WHERE condition, UPDATEs and DELETEs on the underlying table will be restricted to those rows it defines. However UPDATEs may change a row so that it is no longer visible in the view, and an INSERT command can potentiall insert rows which do not satisfy the WHERE condition.

More complex views can be made updatable as before using INSTEAD OF triggers or INSTEAD rules.

Simple example using the following table and view:
<code>
CREATE TABLE postgres_versions (
version VARCHAR(3) PRIMARY KEY,
nickname TEXT NOT NULL
);

INSERT INTO postgres_versions VALUES
('8.0', 'Excitable Element'),
('8.1', 'Fishy Foreign Key'),
('8.2', 'Grumpy Grant'),
('8.3', 'Hysterical Hstore'),
('8.4', 'Insane Index'),
('9.0', 'Jumpy Join'),
('9.1', 'Killer Key'),
('9.2', 'Laconical Lexer'),
('9.3', 'Morose Module');

CREATE VIEW postgres_versions_9 AS
SELECT version, nickname
FROM postgres_versions
WHERE version LIKE '9.%';
</code>

<code>
postgres=# SELECT * from postgres_versions_9;
version | nickname
---------+-----------------
9.0 | Jumpy Join
9.1 | Killer Key
9.2 | Laconical Lexer
9.3 | Morose Module
(4 rows)

postgres=# UPDATE postgres_versions_9 SET nickname='Maniac Master' WHERE version='9.3';
UPDATE 1
postgres=# SELECT * from postgres_versions_9;
version | nickname
---------+-----------------
9.0 | Jumpy Join
9.1 | Killer Key
9.2 | Laconical Lexer
9.3 | Maniac Master
(4 rows)
</code>

'''Links'''
* [http://www.postgresql.org/docs/devel/static/sql-createview.html#SQL-CREATEVIEW-UPDATABLE-VIEWS Documentation]
* [http://www.depesz.com/2012/12/11/waiting-for-9-3-support-automatically-updatable-views/ Waiting for 9.3 – Support automatically-updatable views]
* [http://michael.otacoo.com/postgresql-2/postgres-9-3-feature-highlight-auto-updatable-views/ Postgres 9.3 feature highlight: auto-updatable views]

== Writeable Foreign Tables ==

Foreign data sources can now be written to, as well as read, provided that the FDW driver supports it. As of this writing, the Redis driver supports this, but other drivers are expected to add this enhancement before 9.3 final.

=Backward compatibility=

These changes may incur regressions in your applications.

== CREATE TABLE output ==

CREATE TABLE will no longer output messages about implicit index and sequence creation unless the log level is set to DEBUG1.

== Server settings ==

* Parameter 'commit_delay' is restricted to superusers only
* Parameter 'replication_timeout' has been renamed to 'wal_sender_timeout'
* Parameter 'unix_socket_directory' has been replaced 'unix_socket_directories'
* In-memory sorts to use their full memory allocation; if work_mem was set on the basis of the pre-9.3 behavior, its value may need to be reviewed.

== WAL filenames may end in FF ==

WAL files will now be written in a continuous stream, rather than skipping the last 16MB segment every 4GB, meaning WAL filenames may end in FF. WAL backup or restore scripts may need to be adapted.

PgCon2013CanadaClusterSummit

Kgrittn — Wed, 13 Feb 2013 10:56:26 GMT

Kgrittn: /* RSVP List */ Add Kevin Grittner

= Clustering and Replication Developers Summit pgCon 2013 =

Tuesday, May 21st

9:30AM to 2:30pm

Followed by PostgresXC Summit 3pm to 6pm

University of Ottawa

Room TBD

'''Sponsored by NTT Open Source'''

=== Agenda ===

==== 9AM to 9:30AM ====

Seating and coffee. Please bring any last-minute agenda items to Josh Berkus at this time.

==== 9:30AM to 10:15AM ====

Introductions, and status reports from Replication/Clustering Projects:

Status updates:

* pgPoolII: Tatsuo Ishii
* Postgres-XC: Koichi Suzuki
* Built-in Replication: Simon Riggs

If you are at the summit representing a specific replication or clustering tool, you should prepare a 1-3 minute summary of current progress and issues. If you want to use slides, please provide slides in PDF form to Josh Berkus by Friday, May 17.

==== 10:15AM to 10:30 AM ====

Break

==== 10:30AM to 11:30AM ====

Summary of Clustering API projects.

Summit attendees who have been working on [[ClusterFeatures|core clustering features]] should give an update as to progress and current issues. Please present a 5-10 minute summary. Attendees may use their own laptops for slides, or give slides to Josh Berkus.

Topics:

* Event Triggers:
* Exportable Snapshots:
*

==== 11:30AM to 12:30PM ====

Discussion of priorities, progress and ideas for core clustering projects and APIs.

Goal of this discussion is to modify the list of core clustering features and get commitments for hackers to work on specific features. Also, to supply discussion items for the following day's Developer Meeting

==== 12:30PM to 1:30PM ====

Lunch and follow-up discussion. Box lunches will be supplied.

==== 1:30PM to 2:30PM ====

Follow-up discussion. Consolidation of future clustering API goals and projects.

=== RSVP List ===

# Josh Berkus
# Koichi Suzuki
# Kevin Grittner

= PostgresXC Summit =

3pm to 6pm

Same room as clustering summit.

Agenda TBD.

[[Category:PostgreSQL Events]]

User:Kgrittn

Kgrittn — Thu, 31 Jan 2013 20:29:36 GMT

Kgrittn: /* Bio */ Replace with new bio based on job at EDB

Information about Kevin Grittner and his activities.

[[image:Kevin-Grittner.jpg]]

== Bio ==

I've been making my living working with computers since 1972. In 1980 I founded a consulting company I ran until 2005; during that time I worked with a wide variety of applications, organizations, and technical environments. I'm now with EnterpriseDB as a database architect. I'm a committer for the free, open source version of PostgreSQL, and an active member of the community.

By far the largest patch I've worked on for PostgreSQL has been the Serializable Snapshot Isolation (SSI) implementation which went into PostgreSQL version 9.1. This was a joint project with Dan R.K. Ports of MIT, with helpful input and support from many in the PostgreSQL community. Our paper on that effort can be found here:

http://vldb.org/pvldb/vol5/p1850_danrkports_vldb2012.pdf

For the PostgreSQL docs on the feature, see:

http://www.postgresql.org/docs/current/interactive/transaction-iso.html

For a number of examples see:

http://wiki.postgresql.org/wiki/SSI

== Current Work In Process ==

=== Declarative materialized views ===

For the 9.3 release.

== Possible Future Work ==

=== Rewrite tsearch parser to use regular expressions ===

In reviewing a patch to fix some performance problems in the current parser, I became interested in the possibility of rewriting the current state machine implementation with a regular expression implementation.

[http://archives.postgresql.org/message-id/200912102005.16560.andres@anarazel.de Re: tsearch parser inefficiency if text includes urls or emails - new version]

[http://archives.postgresql.org/message-id/4B210D9E020000250002D344@gw.wicourts.gov tsearch parser overhaul]

=== Deleted WAL files held open by backends in Linux ===

I wasted some time tracking down an oddity which is more of an annoyance in system administration than a real problem, but might look at cleaning it up to save others the bother of investigating the issue.

[http://archives.postgresql.org/message-id/15412.1259630304@sss.pgh.pa.us Deleted WAL files held open by backends in Linux]

=== Temporal data improvements ===

Temporal data handling is weak in SQL in general, and the PostgreSQL implementation seems skewed toward scientific or engineering applications, leaving it weaker than some databases on business applications. (One example would be clean handling of monthy payment schedules.) Enhancements to this area might be interesting.

[http://archives.postgresql.org/message-id/4AE944BD.90809@comcast.net Proposal - temporal contrib module]

[http://archives.postgresql.org/message-id/48692c2d0911171231h6ab16a64yc4db35a6e26909e0@mail.gmail.com Re: Timezones (in 8.5?)]

=== LSB script compliance ===

I came up with a pretty LSB compliant script for Linux; however, the community feels that most of the logic handled in the shell in this script should be moved into pg_ctl. There's a pretty long and winding thread on the topic. The last version of the script might be useful to identify what issues need to be covered in the current pg_ctl code.

[http://archives.postgresql.org/message-id/4A8C41EC.3080708@agliodbs.com We should Axe /contrib/start-scripts]

[http://archives.postgresql.org/message-id/4A9581A5020000250002A37A@gw.wicourts.gov Linux LSB init script]

=== TOAST improvements ===

There have been a few posts to the lists about specific use cases where TOAST defaults are far from optimal. Allowing some tuning here might be helpful.

[http://archives.postgresql.org/message-id/4A6088D50200002500028940@gw.wicourts.gov Higher TOAST compression]

=== Literal and NULL handling anomalies ===

There are a few corner cases where differences between standard and PostgreSQL behaviors with string literals or NULLs astonish those new to PostgreSQL. These aren't easily solved, but might be worth the effort.

=== README files ===

Language is not always as readable as it could be, and some files still largely read like proposals for features which are now implemented.

Server Configuration

Kgrittn — Sun, 20 Jan 2013 14:51:17 GMT

Kgrittn: Use two separate SELECT queries rather than a UNION.

This shows all of the server configuration changes made via updates to the postgresql.conf file, from a running server:
{{SnippetInfo|Dependency display|lang=SQL|category=Administrative}}
<source lang="sql">
SELECT version();
SELECT name, current_setting(name), source
FROM pg_settings
WHERE source NOT IN ('default', 'override');
</source>
[[Category:SQL]]

Server Configuration

Kgrittn — Sat, 19 Jan 2013 17:55:48 GMT

Kgrittn: Add source and use NOT IN ('default', 'override') instead of explicit list of excluded settings, per recommendation of Tom Lane.

This shows all of the server configuration changes made via updates to the postgresql.conf file, from a running server:
{{SnippetInfo|Dependency display|lang=SQL|category=Administrative}}
<source lang="sql">
SELECT 'version'::text AS name, version() AS current_setting, 'version()'::text as source
UNION ALL
SELECT name, current_setting(name), source
FROM pg_settings
WHERE source NOT IN ('default', 'override');
</source>
[[Category:SQL]]

RRReviewers

Kgrittn — Tue, 26 Jun 2012 19:50:37 GMT

Kgrittn: /* What do I need to do to become an RRR? */ Delete reference to obsolete link.

== What's a Round Robin Reviewer (RRR)? ==

Round Robin Reviewers are hackers who volunteer to be assigned random patches (usually according to their level of ability) to review during a commitfest by the [[Running a CommitFest|CommitFest Manager]]. They generally have at least good C skills and a general knowledge of PostgreSQL, but some are accomplished PostgreSQL hackers.

== What do I need to do to become an RRR? ==

# Subscribe to the [http://archives.postgresql.org/pgsql-hackers/ pgsql-hackers] and the [http://archives.postgresql.org/pgsql-rrreviewers/ pgsql-rrreviewers] mailing lists.
# Volunteer to review on pgsql-rrreviewers.
# Sign up for an account on this wiki.
# Read the [[Reviewing a Patch]] page, and the pages & docs they links to.

== What happens next? ==

# The CommitFest Manager will assign you a patch by e-mail.
# Visit [https://commitfest.postgresql.org/ commitfest.postgresql.org], edit the patch, and put your name down as a reviewer. This will confirm that you have set up the community account needed to make entries into the web app, and that you have accepted the assigned patch.
# If you can't or don't want to review that patch, you should reject it within 24 hours (and you will be assigned a new patch if appropriate).
# Read the information on the patch and search the mailing list archives for other comments related to the patch. If you find threads related to the patch which are not yet referenced from the web app entry, please add Comment entries with the ID of a message from the thread.
# Review the patch within 4 days.
# Reply to mail thread on -hackers with comments; make sure the submitter is cc'd.
# Put a summary of your comments on commitfest.postgresql.org by clicking on the patch, setting Comment Type to Review, pasting in the message ID of your email to the -hackers list, and putting in a brief (one or two line) summary of the review.
# Unless this was a preliminary review, and you intend to post an additional review very soon (perhaps after a brief on-list discussion), the status of the patch should change:
## If the patch needs work, and it seems reasonable that the author could complete that work within the current CommitFest, change the status to "Waiting on Author".
## If the patch seems to you ready to accept, change the status to "Ready for Committer".
## If the purpose of the patch is something the community wants, and the overall approach is viable, but more work is needed than is reasonable to expect that the author can complete within the CF, change the status to "Returned with Feedback" and enter "Date Closed".
## If the purpose of the patch is not something the community wants, or the overall approach is not appropriate, change the status to "Rejected" and enter "Date Closed".
# In any case, please e-mail the CommitFest Manager that you are ready to work with another patch.

Note that, if you are assigned a "WIP" patch, it is unlikely to be ready for committing. Instead, you should test the patch as best you can, and report on it to pgsql-hackers.

== Guidelines for Review ==

* Be polite to the patch submitters. You'll probably be one yourself someday.
* Raise ''any'' questions you have on pgsql-hackers mailing list.
* Don't hesitate to ask for help; a poorly done review will slow down the CommitFest far more than not reviewing at all.
* You are helping out in order to reduce the workload on the committers. Do as much as you can, then stop.
* Contact the CommitFest Manager immediately if you won't be able to complete a review.
* When reading from the mail archives, make sure to read the ''entire'' thread about the patch.

[[Category:CommitFest]]

Committing with Git

Kgrittn — Mon, 11 Jun 2012 16:46:03 GMT

Kgrittn: /* Dependent Clone per Branch, Pushing and Pulling From a Local Repository */ Local clone must fetch from master, not pull.

This document is intended for PostgreSQL project [[Committers]]. Regular project contributors should see the introductions for [[Submitting a Patch]]. This is not a complete tutorial on using git. See [[Working with Git]] or the [http://git-scm.com/documentation git documentation].

= Common setup =

1. To connect to the gitmaster repository, you will need to have your PostgreSQL SSH key loaded into your SSH agent, or available somewhere that SSH knows to look for it, such as <tt>~/.ssh/authorized_keys</tt>.

2. Since merge commits may not be pushed, it is a good idea to set up your repository to rebase rather than merging. Without this, if someone else pushes a commit after you pull and before you push, your local repository will contain a merge commit that you'll need to manually remove before you can push.

cd postgresql
git config branch.master.rebase true
git config branch.autosetuprebase always

The last of these commands ensures that any subsequent tracking branches that you create will have branch.<name>.rebase configured automatically.

3. Any commits you push must have matching author and committer tags, and your name and email address must match those configured on the server. So, if you are <tt>Foo Bar <fbar@postgresql.org></tt>, do:

git config user.name "Foo Bar"
git config user.email fbar@postgresql.org

If you use the same email address for all of the git repositories where you commit, you can (or perhaps already have) configure it globally instead; this will update <tt>~/.gitconfig</tt> rather than the <tt>.git/config</tt> for the current repository:

git config --global user.name "Foo Bar"
git config --global user.email foo@bar.net

4. Always use "git push --dry-run" option before the real thing!

=Committing Using a Single Clone=

1. Clone the master repository. Note that this is not the same as the public repository; however, changes from the master repository are regularly pushed to the public repository.

git clone ssh://git@gitmaster.postgresql.org/postgresql.git

2. To commit a patch to the master branch (the equivalent of CVS HEAD), you can use any of the normal git commands. For example, if you've manually modified files or applied a patch from the mailing list that modifies existing files but does not create any new ones, you can just do:

git commit -a

If you've added new files, you must <tt>git add</tt> them first.

git add file1 file2 file3
git commit -a

Or, if the changes you want to commit are on a local branch, you can collapse the commits on the branch into a single commit on the tracking branch using:

git merge --squash branchname

Make sure to use <tt>--squash</tt>, or you'll end up with a merge commit.

3. To back-patch, you can check out the appropriate branch; a local tracking branch will automatically be created. For example:

git checkout REL9_0_STABLE
...hack, hack...
git commit -a

4. Finally, you must push your changes back to the server.

git push

This will push changes in all branches you've updated, but only branches that also exist on the remote side will be pushed; thus, you can have local working branches that won't be pushed. Or, for the avoidance of error, you can configure your repository to push only the current branch:

git config push.default tracking

5. To pull down changes others have committed, you can of course use:

git pull

If you have unpushed changes to any of the branches that have changed on the server, then (1) git will automatically attempt to rebase the currently checked-out branch (because of the configuration you did in step #2) and (2) each other branch that needs to be rebased will be out-of-sync with the server. The easiest way to fix this is to just check out the offending branch and re-pull, e.g.

git checkout REL9_0_STABLE
git pull

6. If one of your tracking branches gets messed up somehow (e.g. you accidentally merge into it, or commit something with the wrong author name/tag) and you can't figure out how to fix it, you can just snap it back to the state in which it exists on the master, throwing away your local changes, e.g.

git checkout master
git reset --hard origin/master

Make sure to use the same branch name in both commands.

=Committing Using Multiple Clones=

When applying a patch to many branches, it can become tedious to keep switching branches; it can be nice to be able to see all the different versions side by side. Furthermore, the PostgreSQL build system isn't smart enough to do the right thing if you switch major releases without cleaning out all the intermediates, so if you do switch branches you'll need to do a complete rebuild each time. (<tt>git clean -dfx</tt> is a useful way to clean all the cruft out of your repository, but be careful that you don't have any untracked files there that you meant to keep.)

The use <tt>git clone --reference</tt> is not recommended except for short-term, throw-away copies, because a subsequent <tt>git gc</tt> can result in data loss. Instead, use one of the techniques described below.

==Independent Clone per Branch==

The simplest way to commit using multiple clones is to create them as described above for a single clone, and keep a different branch checked out in each copy. You can configure each clone to pull only the branch you care about for that clone. For example, for the REL9_0_STABLE branch:

git clone ssh://git@gitmaster.postgresql.org/postgresql.git REL9_0_STABLE
cd REL9_0_STABLE
git checkout REL9_0_STABLE
git config branch.REL9_0_STABLE.rebase true
git config user.name "Foo Bar"
git config user.email fbar@postgresql.org
git config remote.origin.fetch '+refs/heads/REL9_0_STABLE:refs/remotes/origin/REL9_0_STABLE'
git branch -D master

One disadvantage of this approach is that you will use more disk space: the <tt>.git</tt> directory for each repository, as of this writing, is a bit more than 220MB. If this is a concern, use one of the methods described below.

==Dependent Clone per Branch, Pushing and Pulling From a Local Repository==

Git will automatically use hard links when cloning a repository stored on the local machine. So, you could do this:

git clone --bare --mirror ssh://git@gitmaster.postgresql.org/postgresql.git
git clone postgresql REL9_0_STABLE
cd REL9_0_STABLE
git checkout REL9_0_STABLE
git config branch.REL9_0_STABLE.rebase true
git config user.name "Foo Bar"
git config user.email fbar@postgresql.org
git config remote.origin.fetch '+refs/heads/REL9_0_STABLE:refs/remotes/origin/REL9_0_STABLE'
git branch -D master

All of these steps except the first should be repeated for each branch for which you wish to maintain a clone. (If your user name and/or email are configured globally, you need not configure them again for each new repository.)

With this approach, the REL9_0_STABLE repository will pull from and push to the local postgresql repository, which in turn will pull from and push to the master server. Thus, you must do this to update (supposing both repositories are in your home directory):

cd ~/postgresql.git
git fetch
cd ~/REL9_0_STABLE
git pull

And to push, you must do this:

cd ~/REL9_0_STABLE
git push
cd ~/postgresql.git
git push

It would probably be wise to script this, if you plan to do it regularly and with multiple branches.

==Clone Locally, Repoint Origin==

The solution described in the previous section can be inconvenient, since it requires pushing and pulling each commit twice. One possible way to avoid this is to create a single clone from the master, then clone it multiple times locally, then repoint the origin server for each such local clone at the master. The history existing at the time of the initial clone will be shared among all the local clones (using hard links), but any new history fetched after the initial setup will consume separate storage in each local clone. This still represents a substantial savings in disk space, while avoiding the inconvenience of pushing and pulling twice.

To do this, set up each clone as described in the previous section and then perform the following additional steps:

git remote set-url origin ssh://git@gitmaster.postgresql.org/postgresql.git
git remote update
git remote prune origin

==Dependent Clone per Branch, Pulling From a Local Repository and Pushing to the Remote Repository==

This method is like the "Dependent Clone per Branch, Pushing and Pulling From a Local Repository" recipe, but instead of pushing back to the local repository you set it up push direct to the remote repository, by doing, in each clone of your local mirror:

git remote set-url --push origin ssh://git@gitmaster.postgresql.org/postgresql.git

This requires a fairly modern version of git.

You can also avoid having to fetch into the mirror and then pull into the clone in separate steps by using a shell alias. Here is a function that works with bash to combine these steps:

function pgpull ()
{
pushd /path/to/mirror > /dev/null
git fetch
popd > /dev/null
git pull
}

= Committing Using a Single Clone and multiple workdirs =

This method is similar to the method with single clone, but you can keep each active branch checked out all the time.

1. Create a directory to hold all the cloned repository and all the workdirs (makes it easier to remember that they're all linked to the same clone).

mkdir pgsql-git; cd pgsql-git

2. Clone the master repository. Note that this is not the same as the public repository; however, changes from the master repository are regularly pushed to the public repository.

git clone ssh://git@gitmaster.postgresql.org/postgresql.git

This creates a directory called "postgresql", and the master branch is automatically checked out into that working directory. The cloned git repository is in "postgresql/.git", which is shared with all the other workdirs we create later.

3. Prevent automatic git garbage collection.

git --git-dir=postgresql/.git/ config gc.auto 0

Rationale: With this method, you have multiple checkouts from a single repository, but "git gc" does not know about the other working directories. That is not a problem in general, but if you run "git gc" when you have staged but uncommitted in a workdir other than the master one, those changes can be lost. This is a known limitation with <tt>git-new-workdir</tt>, see [http://kerneltrap.org/mailarchive/git/2007/10/11/335637]. Make sure you don't run "git gc" when all the back-branch checkouts are not in a clean state, and you should be safe.

4. Create workdirs for all active backbranches. The <tt>git-new-workdir</tt> tool is in git contrib directory, the path in the example below is for Debian:

sh /usr/share/doc/git/contrib/workdir/git-new-workdir postgresql/.git/ 90stable
cd 90stable
git checkout -b REL9_0_STABLE origin/REL9_0_STABLE
cd ..

<repeat for every active back-branch>

5. You can now work separately on the checkouts as you would on CVS. All the local branches, tracking the corresponding remote branches, are in the repository shared by all workdirs. To commit a patch to any of the branches, you can use any of the normal git commands. For example, if you've manually modified files or applied a patch from the mailing list that modifies existing files but does not create any new ones, you can just do:

git commit -a

If you've added new files, you must <tt>git add</tt> them first.

git add file1 file2 file3
git commit -a

Or, if the changes you want to commit are on a local branch, you can collapse the commits on the branch into a single commit on the tracking branch using:

git merge --squash branchname

Make sure to use <tt>--squash</tt>, or you'll end up with a merge commit.

6. Finally, you must push your changes back to the server. This will push changes in *all* branches you've committed to:

git push --dry-run

This will show what's being pushed without doing anything yet, Double-check the changes, using "git log" and "git diff" with the commitid ranges printed out. Once your satisfied, push them for real:

git push

7. To pull down changes others have committed, you can of course use:

git pull

If you have unpushed changes to any of the branches that have changed on the server, then (1) git will automatically attempt to rebase the currently checked-out branch (because of the configuration you did earlier) and (2) each other branch that needs to be rebased will be out-of-sync with the server. The easiest way to fix this is to just check out the offending branch and re-pull, e.g.

git checkout REL9_0_STABLE
git pull

8. If one of your tracking branches gets messed up somehow (e.g. you accidentally merge into it, or commit something with the wrong author name/tag) and you can't figure out how to fix it, you can just snap it back to the state in which it exists on the master, throwing away your local changes, e.g.

git checkout master
git reset --hard origin/master

Make sure to use the branch name matching the workdir you're in in both commands.

= Making a new release branch on origin =

To create a new branch in the gitmaster repo starting from the current tip of ''master'', do this:

git pull # be sure you have the latest "master"
git push origin master:refs/heads/"new-branch-name"

for example

git push origin master:refs/heads/REL9_1_STABLE

After this, check out the branch locally following whichever of the previous arrangements you are using.

By convention, only release branches should be pushed to the gitmaster repo; don't push experimental or feature branches there.

[[Category:Git]]

Number Of Database Connections

Kgrittn — Thu, 17 May 2012 10:48:44 GMT

Kgrittn: /* How to Find the Optimal Database Connection Pool Size */ Change reference to 9.3 to 9.2.

You can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. This page attempts to explain why that is.

== Summary ==

A database server only has so many resources, and if you don't have enough connections active to use all of them, your throughput will generally improve by using more connections. Once all of the resources are in use, you won't push any more work through by having more connections competing for the resources. In fact, throughput starts to fall off due to the overhead from that contention. You can generally improve both latency and throughput by limiting the number of database connections with active transactions to match the available number of resources, and queuing any requests to start a new database transaction which come in while at the limit.

Contrary to many people's initial intuitive impulses, you will often see a transaction ''reach completion sooner'' if you queue it when it is ready but the system is busy enough to have saturated resources and ''start it later'' when resources become available.

== The Need for an External Pool ==

If you look at any graph of PostgreSQL performance with number of connections on the ''x'' axis and tps on the ''y'' access (with nothing else changing), you will see performance climb as connections rise until you hit saturation, and then you have a "knee" after which performance falls off. A lot of work has been done for version 9.2 to push that knee to the right and make the fall-off more gradual, but the issue is intrinsic -- without a built-in connection pool or at least an admission control policy, the knee and subsequent performance degradation will always be there.

The decision not to include a connection pooler inside the PostgreSQL server itself has been taken deliberately and with good reason. In many cases you will get better performance if the connection pooler is running on a separate machine. Also, you can get improved functionality by incorporating a connection pool into client-side software. Many frameworks do the pooling in a process running on the the database server machine (to minimize latency effects from the database protocol) and accept high-level requests to run a certain function with a given set of parameters, with the entire function running as a single database transaction. This ensures that network latency or connection failures can't cause a transaction to hang while waiting for something from the network, and provides a simple way to retry any database transaction which rolls back with a serialization failure (SQLSTATE 40001 or 40P01).

Since a pooler built in to the database engine would be inferior (for the above reasons), the community has decided not to go that route.

== Reasons for Performance Reduction Past the "Knee" ==

There are a number of independent reasons that performance falls off with more database connections.

* '''Context switches'''. The processor is interrupted from working on one query and has to switch to another, which involves saving state and restoring state. While the core is busy swapping states it is not doing any useful work on any query.

* '''Cache line contention'''. One query is likely to be working on a particular area of RAM, and the query taking its place is likely to be working on a different area; causing data cached on the CPU chip to be discarded, only to need to be reloaded to continue the other query. Besides that the various processes will be grabbing control of cache lines from each other, causing stalls. (Humorous note, in one oprofile run of a heavily contended load, 10% of CPU time was attributed to a 1-byte noop; analysis showed that it was because it needed to wait on a cache line for the following machine code operation.)

* '''Lock contention'''. This happens at various levels: spinlocks, LW locks, and all the locks that show up in pg_locks. As more processes compete for the spinlocks (which protect LW locks acquisition and release, which in turn protect the heavyweight and predicate lock acquisition and release) they account for a high percentage of CPU time used.

* '''RAM usage'''. The work_mem setting can have a big impact on performance. If it is too small, hash tables and sorts spill to disk, bitmap heap scans become "lossy", requiring more work on each page access, etc. So you want it to be big. But work_mem RAM can be allocated for each node of a query on each connection, all at the same time. So a big work_mem with a large number of connections can cause a lot of the OS cache to be periodically discarded, forcing more accesses to disk; or it could even put the system into swapping. So the more connections you have, the more you need to make a choice between slow plans and trashing cache/swapping.

* '''Disk access'''. If you ''do'' need to go to disk for random access, a large number of connections can tend to force more tables and indexes to be accessed at the same time, causing heavier seeking all over the disk.

* '''General scaling'''. Some internal structures allocated based on max_connections scale at O(N^2) or O(N*log(N)). Some types of overhead which are negligible at a lower number of connections can become significant with a large number of connections.

== How to Find the Optimal Database Connection Pool Size ==

A formula which has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((''core_count'' * 2) + ''effective_spindle_count''). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. Benchmarks of WIP for version 9.2 suggest that this formula will need adjustment on that release. There hasn't been any analysis so far regarding how well the formula works with SSDs.

However you choose a starting point for a connection pool size, you should probably try incremental adjustments with your production system to find the actual "sweet spot" for your hardware and workload.

Number Of Database Connections

Kgrittn — Thu, 17 May 2012 10:48:02 GMT

Kgrittn: /* The Need for an External Pool */ Fix reference to 9.3 to say 9.2.

You can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. This page attempts to explain why that is.

== Summary ==

A database server only has so many resources, and if you don't have enough connections active to use all of them, your throughput will generally improve by using more connections. Once all of the resources are in use, you won't push any more work through by having more connections competing for the resources. In fact, throughput starts to fall off due to the overhead from that contention. You can generally improve both latency and throughput by limiting the number of database connections with active transactions to match the available number of resources, and queuing any requests to start a new database transaction which come in while at the limit.

Contrary to many people's initial intuitive impulses, you will often see a transaction ''reach completion sooner'' if you queue it when it is ready but the system is busy enough to have saturated resources and ''start it later'' when resources become available.

== The Need for an External Pool ==

If you look at any graph of PostgreSQL performance with number of connections on the ''x'' axis and tps on the ''y'' access (with nothing else changing), you will see performance climb as connections rise until you hit saturation, and then you have a "knee" after which performance falls off. A lot of work has been done for version 9.2 to push that knee to the right and make the fall-off more gradual, but the issue is intrinsic -- without a built-in connection pool or at least an admission control policy, the knee and subsequent performance degradation will always be there.

The decision not to include a connection pooler inside the PostgreSQL server itself has been taken deliberately and with good reason. In many cases you will get better performance if the connection pooler is running on a separate machine. Also, you can get improved functionality by incorporating a connection pool into client-side software. Many frameworks do the pooling in a process running on the the database server machine (to minimize latency effects from the database protocol) and accept high-level requests to run a certain function with a given set of parameters, with the entire function running as a single database transaction. This ensures that network latency or connection failures can't cause a transaction to hang while waiting for something from the network, and provides a simple way to retry any database transaction which rolls back with a serialization failure (SQLSTATE 40001 or 40P01).

Since a pooler built in to the database engine would be inferior (for the above reasons), the community has decided not to go that route.

== Reasons for Performance Reduction Past the "Knee" ==

There are a number of independent reasons that performance falls off with more database connections.

* '''Context switches'''. The processor is interrupted from working on one query and has to switch to another, which involves saving state and restoring state. While the core is busy swapping states it is not doing any useful work on any query.

* '''Cache line contention'''. One query is likely to be working on a particular area of RAM, and the query taking its place is likely to be working on a different area; causing data cached on the CPU chip to be discarded, only to need to be reloaded to continue the other query. Besides that the various processes will be grabbing control of cache lines from each other, causing stalls. (Humorous note, in one oprofile run of a heavily contended load, 10% of CPU time was attributed to a 1-byte noop; analysis showed that it was because it needed to wait on a cache line for the following machine code operation.)

* '''Lock contention'''. This happens at various levels: spinlocks, LW locks, and all the locks that show up in pg_locks. As more processes compete for the spinlocks (which protect LW locks acquisition and release, which in turn protect the heavyweight and predicate lock acquisition and release) they account for a high percentage of CPU time used.

* '''RAM usage'''. The work_mem setting can have a big impact on performance. If it is too small, hash tables and sorts spill to disk, bitmap heap scans become "lossy", requiring more work on each page access, etc. So you want it to be big. But work_mem RAM can be allocated for each node of a query on each connection, all at the same time. So a big work_mem with a large number of connections can cause a lot of the OS cache to be periodically discarded, forcing more accesses to disk; or it could even put the system into swapping. So the more connections you have, the more you need to make a choice between slow plans and trashing cache/swapping.

* '''Disk access'''. If you ''do'' need to go to disk for random access, a large number of connections can tend to force more tables and indexes to be accessed at the same time, causing heavier seeking all over the disk.

* '''General scaling'''. Some internal structures allocated based on max_connections scale at O(N^2) or O(N*log(N)). Some types of overhead which are negligible at a lower number of connections can become significant with a large number of connections.

== How to Find the Optimal Database Connection Pool Size ==

A formula which has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((''core_count'' * 2) + ''effective_spindle_count''). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. Benchmarks of WIP for version 9.3 suggest that this formula will need adjustment on that release. There hasn't been any analysis so far regarding how well the formula works with SSDs.

However you choose a starting point for a connection pool size, you should probably try incremental adjustments with your production system to find the actual "sweet spot" for your hardware and workload.

Number Of Database Connections

Kgrittn — Fri, 11 May 2012 03:28:33 GMT

Kgrittn: /* How to Find the Optimal Database Connection Pool Size */ Fix typo.

You can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. This page attempts to explain why that is.

== Summary ==

A database server only has so many resources, and if you don't have enough connections active to use all of them, your throughput will generally improve by using more connections. Once all of the resources are in use, you won't push any more work through by having more connections competing for the resources. In fact, throughput starts to fall off due to the overhead from that contention. You can generally improve both latency and throughput by limiting the number of database connections with active transactions to match the available number of resources, and queuing any requests to start a new database transaction which come in while at the limit.

Contrary to many people's initial intuitive impulses, you will often see a transaction ''reach completion sooner'' if you queue it when it is ready but the system is busy enough to have saturated resources and ''start it later'' when resources become available.

== The Need for an External Pool ==

If you look at any graph of PostgreSQL performance with number of connections on the ''x'' axis and tps on the ''y'' access (with nothing else changing), you will see performance climb as connections rise until you hit saturation, and then you have a "knee" after which performance falls off. A lot of work has been done for version 9.3 to push that knee to the right and make the fall-off more gradual, but the issue is intrinsic -- without a built-in connection pool or at least an admission control policy, the knee and subsequent performance degradation will always be there.

The decision not to include a connection pooler inside the PostgreSQL server itself has been taken deliberately and with good reason. In many cases you will get better performance if the connection pooler is running on a separate machine. Also, you can get improved functionality by incorporating a connection pool into client-side software. Many frameworks do the pooling in a process running on the the database server machine (to minimize latency effects from the database protocol) and accept high-level requests to run a certain function with a given set of parameters, with the entire function running as a single database transaction. This ensures that network latency or connection failures can't cause a transaction to hang while waiting for something from the network, and provides a simple way to retry any database transaction which rolls back with a serialization failure (SQLSTATE 40001 or 40P01).

Since a pooler built in to the database engine would be inferior (for the above reasons), the community has decided not to go that route.

== Reasons for Performance Reduction Past the "Knee" ==

There are a number of independent reasons that performance falls off with more database connections.

* '''Context switches'''. The processor is interrupted from working on one query and has to switch to another, which involves saving state and restoring state. While the core is busy swapping states it is not doing any useful work on any query.

* '''Cache line contention'''. One query is likely to be working on a particular area of RAM, and the query taking its place is likely to be working on a different area; causing data cached on the CPU chip to be discarded, only to need to be reloaded to continue the other query. Besides that the various processes will be grabbing control of cache lines from each other, causing stalls. (Humorous note, in one oprofile run of a heavily contended load, 10% of CPU time was attributed to a 1-byte noop; analysis showed that it was because it needed to wait on a cache line for the following machine code operation.)

* '''Lock contention'''. This happens at various levels: spinlocks, LW locks, and all the locks that show up in pg_locks. As more processes compete for the spinlocks (which protect LW locks acquisition and release, which in turn protect the heavyweight and predicate lock acquisition and release) they account for a high percentage of CPU time used.

* '''RAM usage'''. The work_mem setting can have a big impact on performance. If it is too small, hash tables and sorts spill to disk, bitmap heap scans become "lossy", requiring more work on each page access, etc. So you want it to be big. But work_mem RAM can be allocated for each node of a query on each connection, all at the same time. So a big work_mem with a large number of connections can cause a lot of the OS cache to be periodically discarded, forcing more accesses to disk; or it could even put the system into swapping. So the more connections you have, the more you need to make a choice between slow plans and trashing cache/swapping.

* '''Disk access'''. If you ''do'' need to go to disk for random access, a large number of connections can tend to force more tables and indexes to be accessed at the same time, causing heavier seeking all over the disk.

* '''General scaling'''. Some internal structures allocated based on max_connections scale at O(N^2) or O(N*log(N)). Some types of overhead which are negligible at a lower number of connections can become significant with a large number of connections.

== How to Find the Optimal Database Connection Pool Size ==

A formula which has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((''core_count'' * 2) + ''effective_spindle_count''). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. Benchmarks of WIP for version 9.3 suggest that this formula will need adjustment on that release. There hasn't been any analysis so far regarding how well the formula works with SSDs.

However you choose a starting point for a connection pool size, you should probably try incremental adjustments with your production system to find the actual "sweet spot" for your hardware and workload.

Number Of Database Connections

Kgrittn — Thu, 10 May 2012 15:46:53 GMT

Kgrittn: /* Summary */ Add a rough description of the desired pool to the summary, and the counter-intuitive nature of a later start time resulting in earlier completion.

You can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. This page attempts to explain why that is.

== Summary ==

A database server only has so many resources, and if you don't have enough connections active to use all of them, your throughput will generally improve by using more connections. Once all of the resources are in use, you won't push any more work through by having more connections competing for the resources. In fact, throughput starts to fall off due to the overhead from that contention. You can generally improve both latency and throughput by limiting the number of database connections with active transactions to match the available number of resources, and queuing any requests to start a new database transaction which come in while at the limit.

Contrary to many people's initial intuitive impulses, you will often see a transaction ''reach completion sooner'' if you queue it when it is ready but the system is busy enough to have saturated resources and ''start it later'' when resources become available.

== The Need for an External Pool ==

If you look at any graph of PostgreSQL performance with number of connections on the ''x'' axis and tps on the ''y'' access (with nothing else changing), you will see performance climb as connections rise until you hit saturation, and then you have a "knee" after which performance falls off. A lot of work has been done for version 9.3 to push that knee to the right and make the fall-off more gradual, but the issue is intrinsic -- without a built-in connection pool or at least an admission control policy, the knee and subsequent performance degradation will always be there.

The decision not to include a connection pooler inside the PostgreSQL server itself has been taken deliberately and with good reason. In many cases you will get better performance if the connection pooler is running on a separate machine. Also, you can get improved functionality by incorporating a connection pool into client-side software. Many frameworks do the pooling in a process running on the the database server machine (to minimize latency effects from the database protocol) and accept high-level requests to run a certain function with a given set of parameters, with the entire function running as a single database transaction. This ensures that network latency or connection failures can't cause a transaction to hang while waiting for something from the network, and provides a simple way to retry any database transaction which rolls back with a serialization failure (SQLSTATE 40001 or 40P01).

Since a pooler built in to the database engine would be inferior (for the above reasons), the community has decided not to go that route.

== Reasons for Performance Reduction Past the "Knee" ==

There are a number of independent reasons that performance falls off with more database connections.

* '''Context switches'''. The processor is interrupted from working on one query and has to switch to another, which involves saving state and restoring state. While the core is busy swapping states it is not doing any useful work on any query.

* '''Cache line contention'''. One query is likely to be working on a particular area of RAM, and the query taking its place is likely to be working on a different area; causing data cached on the CPU chip to be discarded, only to need to be reloaded to continue the other query. Besides that the various processes will be grabbing control of cache lines from each other, causing stalls. (Humorous note, in one oprofile run of a heavily contended load, 10% of CPU time was attributed to a 1-byte noop; analysis showed that it was because it needed to wait on a cache line for the following machine code operation.)

* '''Lock contention'''. This happens at various levels: spinlocks, LW locks, and all the locks that show up in pg_locks. As more processes compete for the spinlocks (which protect LW locks acquisition and release, which in turn protect the heavyweight and predicate lock acquisition and release) they account for a high percentage of CPU time used.

* '''RAM usage'''. The work_mem setting can have a big impact on performance. If it is too small, hash tables and sorts spill to disk, bitmap heap scans become "lossy", requiring more work on each page access, etc. So you want it to be big. But work_mem RAM can be allocated for each node of a query on each connection, all at the same time. So a big work_mem with a large number of connections can cause a lot of the OS cache to be periodically discarded, forcing more accesses to disk; or it could even put the system into swapping. So the more connections you have, the more you need to make a choice between slow plans and trashing cache/swapping.

* '''Disk access'''. If you ''do'' need to go to disk for random access, a large number of connections can tend to force more tables and indexes to be accessed at the same time, causing heavier seeking all over the disk.

* '''General scaling'''. Some internal structures allocated based on max_connections scale at O(N^2) or O(N*log(N)). Some types of overhead which are negligible at a lower number of connections can become significant with a large number of connections.

== How to Find the Optimal Database Connection Pool Size ==

A formula which has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((''core_count'' * 2) + ''effective_spindle_count''). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. Benchmarks of WIP for version 9.3 suggest that this formula will need adjustment on that release. There hasn't been any analysis so far regarding how well the formula works with SDDs.

However you choose a starting point for a connection pool size, you should probably try incremental adjustments with your production system to find the actual "sweet spot" for your hardware and workload.

Number Of Database Connections

Kgrittn — Thu, 10 May 2012 15:36:36 GMT

Kgrittn: /* How to Find Optimal Database Connection Pool Size */ Add "the" to section title.

You can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. This page attempts to explain why that is.

== Summary ==

A database server only has so many resources, and if you don't have enough connections active to use all of them, your throughput will generally improve by using more connections. Once all of the resources are in use, you won't push any more through by having more connections competing for the resources. In fact, throughput starts to fall off due to the overhead from that contention.

== The Need for an External Pool ==

If you look at any graph of PostgreSQL performance with number of connections on the ''x'' axis and tps on the ''y'' access (with nothing else changing), you will see performance climb as connections rise until you hit saturation, and then you have a "knee" after which performance falls off. A lot of work has been done for version 9.3 to push that knee to the right and make the fall-off more gradual, but the issue is intrinsic -- without a built-in connection pool or at least an admission control policy, the knee and subsequent performance degradation will always be there.

The decision not to include a connection pooler inside the PostgreSQL server itself has been taken deliberately and with good reason. In many cases you will get better performance if the connection pooler is running on a separate machine. Also, you can get improved functionality by incorporating a connection pool into client-side software. Many frameworks do the pooling in a process running on the the database server machine (to minimize latency effects from the database protocol) and accept high-level requests to run a certain function with a given set of parameters, with the entire function running as a single database transaction. This ensures that network latency or connection failures can't cause a transaction to hang while waiting for something from the network, and provides a simple way to retry any database transaction which rolls back with a serialization failure (SQLSTATE 40001 or 40P01).

Since a pooler built in to the database engine would be inferior (for the above reasons), the community has decided not to go that route.

== Reasons for Performance Reduction Past the "Knee" ==

There are a number of independent reasons that performance falls off with more database connections.

* '''Context switches'''. The processor is interrupted from working on one query and has to switch to another, which involves saving state and restoring state. While the core is busy swapping states it is not doing any useful work on any query.

* '''Cache line contention'''. One query is likely to be working on a particular area of RAM, and the query taking its place is likely to be working on a different area; causing data cached on the CPU chip to be discarded, only to need to be reloaded to continue the other query. Besides that the various processes will be grabbing control of cache lines from each other, causing stalls. (Humorous note, in one oprofile run of a heavily contended load, 10% of CPU time was attributed to a 1-byte noop; analysis showed that it was because it needed to wait on a cache line for the following machine code operation.)

* '''Lock contention'''. This happens at various levels: spinlocks, LW locks, and all the locks that show up in pg_locks. As more processes compete for the spinlocks (which protect LW locks acquisition and release, which in turn protect the heavyweight and predicate lock acquisition and release) they account for a high percentage of CPU time used.

* '''RAM usage'''. The work_mem setting can have a big impact on performance. If it is too small, hash tables and sorts spill to disk, bitmap heap scans become "lossy", requiring more work on each page access, etc. So you want it to be big. But work_mem RAM can be allocated for each node of a query on each connection, all at the same time. So a big work_mem with a large number of connections can cause a lot of the OS cache to be periodically discarded, forcing more accesses to disk; or it could even put the system into swapping. So the more connections you have, the more you need to make a choice between slow plans and trashing cache/swapping.

* '''Disk access'''. If you ''do'' need to go to disk for random access, a large number of connections can tend to force more tables and indexes to be accessed at the same time, causing heavier seeking all over the disk.

* '''General scaling'''. Some internal structures allocated based on max_connections scale at O(N^2) or O(N*log(N)). Some types of overhead which are negligible at a lower number of connections can become significant with a large number of connections.

== How to Find the Optimal Database Connection Pool Size ==

A formula which has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((''core_count'' * 2) + ''effective_spindle_count''). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. Benchmarks of WIP for version 9.3 suggest that this formula will need adjustment on that release. There hasn't been any analysis so far regarding how well the formula works with SDDs.

However you choose a starting point for a connection pool size, you should probably try incremental adjustments with your production system to find the actual "sweet spot" for your hardware and workload.

Number Of Database Connections

Kgrittn — Thu, 10 May 2012 15:35:42 GMT

Kgrittn: /* Reasons for Performance Reduction Past "Knee" */ add "the" to section title

You can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. This page attempts to explain why that is.

== Summary ==

A database server only has so many resources, and if you don't have enough connections active to use all of them, your throughput will generally improve by using more connections. Once all of the resources are in use, you won't push any more through by having more connections competing for the resources. In fact, throughput starts to fall off due to the overhead from that contention.

== The Need for an External Pool ==

If you look at any graph of PostgreSQL performance with number of connections on the ''x'' axis and tps on the ''y'' access (with nothing else changing), you will see performance climb as connections rise until you hit saturation, and then you have a "knee" after which performance falls off. A lot of work has been done for version 9.3 to push that knee to the right and make the fall-off more gradual, but the issue is intrinsic -- without a built-in connection pool or at least an admission control policy, the knee and subsequent performance degradation will always be there.

The decision not to include a connection pooler inside the PostgreSQL server itself has been taken deliberately and with good reason. In many cases you will get better performance if the connection pooler is running on a separate machine. Also, you can get improved functionality by incorporating a connection pool into client-side software. Many frameworks do the pooling in a process running on the the database server machine (to minimize latency effects from the database protocol) and accept high-level requests to run a certain function with a given set of parameters, with the entire function running as a single database transaction. This ensures that network latency or connection failures can't cause a transaction to hang while waiting for something from the network, and provides a simple way to retry any database transaction which rolls back with a serialization failure (SQLSTATE 40001 or 40P01).

Since a pooler built in to the database engine would be inferior (for the above reasons), the community has decided not to go that route.

== Reasons for Performance Reduction Past the "Knee" ==

There are a number of independent reasons that performance falls off with more database connections.

* '''Context switches'''. The processor is interrupted from working on one query and has to switch to another, which involves saving state and restoring state. While the core is busy swapping states it is not doing any useful work on any query.

* '''Cache line contention'''. One query is likely to be working on a particular area of RAM, and the query taking its place is likely to be working on a different area; causing data cached on the CPU chip to be discarded, only to need to be reloaded to continue the other query. Besides that the various processes will be grabbing control of cache lines from each other, causing stalls. (Humorous note, in one oprofile run of a heavily contended load, 10% of CPU time was attributed to a 1-byte noop; analysis showed that it was because it needed to wait on a cache line for the following machine code operation.)

* '''Lock contention'''. This happens at various levels: spinlocks, LW locks, and all the locks that show up in pg_locks. As more processes compete for the spinlocks (which protect LW locks acquisition and release, which in turn protect the heavyweight and predicate lock acquisition and release) they account for a high percentage of CPU time used.

* '''RAM usage'''. The work_mem setting can have a big impact on performance. If it is too small, hash tables and sorts spill to disk, bitmap heap scans become "lossy", requiring more work on each page access, etc. So you want it to be big. But work_mem RAM can be allocated for each node of a query on each connection, all at the same time. So a big work_mem with a large number of connections can cause a lot of the OS cache to be periodically discarded, forcing more accesses to disk; or it could even put the system into swapping. So the more connections you have, the more you need to make a choice between slow plans and trashing cache/swapping.

* '''Disk access'''. If you ''do'' need to go to disk for random access, a large number of connections can tend to force more tables and indexes to be accessed at the same time, causing heavier seeking all over the disk.

* '''General scaling'''. Some internal structures allocated based on max_connections scale at O(N^2) or O(N*log(N)). Some types of overhead which are negligible at a lower number of connections can become significant with a large number of connections.

== How to Find Optimal Database Connection Pool Size ==

A formula which has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((''core_count'' * 2) + ''effective_spindle_count''). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. Benchmarks of WIP for version 9.3 suggest that this formula will need adjustment on that release. There hasn't been any analysis so far regarding how well the formula works with SDDs.

However you choose a starting point for a connection pool size, you should probably try incremental adjustments with your production system to find the actual "sweet spot" for your hardware and workload.

Number Of Database Connections

Kgrittn — Thu, 10 May 2012 15:34:36 GMT

Kgrittn: Initial page creation

You can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. This page attempts to explain why that is.

== Summary ==

A database server only has so many resources, and if you don't have enough connections active to use all of them, your throughput will generally improve by using more connections. Once all of the resources are in use, you won't push any more through by having more connections competing for the resources. In fact, throughput starts to fall off due to the overhead from that contention.

== The Need for an External Pool ==

If you look at any graph of PostgreSQL performance with number of connections on the ''x'' axis and tps on the ''y'' access (with nothing else changing), you will see performance climb as connections rise until you hit saturation, and then you have a "knee" after which performance falls off. A lot of work has been done for version 9.3 to push that knee to the right and make the fall-off more gradual, but the issue is intrinsic -- without a built-in connection pool or at least an admission control policy, the knee and subsequent performance degradation will always be there.

The decision not to include a connection pooler inside the PostgreSQL server itself has been taken deliberately and with good reason. In many cases you will get better performance if the connection pooler is running on a separate machine. Also, you can get improved functionality by incorporating a connection pool into client-side software. Many frameworks do the pooling in a process running on the the database server machine (to minimize latency effects from the database protocol) and accept high-level requests to run a certain function with a given set of parameters, with the entire function running as a single database transaction. This ensures that network latency or connection failures can't cause a transaction to hang while waiting for something from the network, and provides a simple way to retry any database transaction which rolls back with a serialization failure (SQLSTATE 40001 or 40P01).

Since a pooler built in to the database engine would be inferior (for the above reasons), the community has decided not to go that route.

== Reasons for Performance Reduction Past "Knee" ==

There are a number of independent reasons that performance falls off with more database connections.

* '''Context switches'''. The processor is interrupted from working on one query and has to switch to another, which involves saving state and restoring state. While the core is busy swapping states it is not doing any useful work on any query.

* '''Cache line contention'''. One query is likely to be working on a particular area of RAM, and the query taking its place is likely to be working on a different area; causing data cached on the CPU chip to be discarded, only to need to be reloaded to continue the other query. Besides that the various processes will be grabbing control of cache lines from each other, causing stalls. (Humorous note, in one oprofile run of a heavily contended load, 10% of CPU time was attributed to a 1-byte noop; analysis showed that it was because it needed to wait on a cache line for the following machine code operation.)

* '''Lock contention'''. This happens at various levels: spinlocks, LW locks, and all the locks that show up in pg_locks. As more processes compete for the spinlocks (which protect LW locks acquisition and release, which in turn protect the heavyweight and predicate lock acquisition and release) they account for a high percentage of CPU time used.

* '''RAM usage'''. The work_mem setting can have a big impact on performance. If it is too small, hash tables and sorts spill to disk, bitmap heap scans become "lossy", requiring more work on each page access, etc. So you want it to be big. But work_mem RAM can be allocated for each node of a query on each connection, all at the same time. So a big work_mem with a large number of connections can cause a lot of the OS cache to be periodically discarded, forcing more accesses to disk; or it could even put the system into swapping. So the more connections you have, the more you need to make a choice between slow plans and trashing cache/swapping.

* '''Disk access'''. If you ''do'' need to go to disk for random access, a large number of connections can tend to force more tables and indexes to be accessed at the same time, causing heavier seeking all over the disk.

* '''General scaling'''. Some internal structures allocated based on max_connections scale at O(N^2) or O(N*log(N)). Some types of overhead which are negligible at a lower number of connections can become significant with a large number of connections.

== How to Find Optimal Database Connection Pool Size ==

A formula which has held up pretty well across a lot of benchmarks for years is that for optimal throughput the number of active connections should be somewhere near ((''core_count'' * 2) + ''effective_spindle_count''). Core count should not include HT threads, even if hyperthreading is enabled. Effective spindle count is zero if the active data set is fully cached, and approaches the actual number of spindles as the cache hit rate falls. Benchmarks of WIP for version 9.3 suggest that this formula will need adjustment on that release. There hasn't been any analysis so far regarding how well the formula works with SDDs.

However you choose a starting point for a connection pool size, you should probably try incremental adjustments with your production system to find the actual "sweet spot" for your hardware and workload.

PgCon 2012 Developer Meeting

Kgrittn — Wed, 09 May 2012 21:48:07 GMT

Kgrittn: /* Agenda */ Correct apparent typo in agenda item for CF schedule: we want to discuss 9.3 CF schedule, not 9.2

A meeting of the most active PostgreSQL developers is being planned for Wednesday 16th May, 2012 near the University of Ottawa, prior to pgCon 2012. In order to keep the numbers manageable, this meeting is '''by invitation only'''. Unfortunately it is quite possible that we've overlooked important code developers during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that this year the attendee numbers have been cut to try to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.2 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies, unlike in previous years.

This is a PostgreSQL Community event. Room and refreshments/food sponsored by EnterpriseDB. Other companies sponsored attendance for their developers.

== Time & Location ==

The meeting will be from 9AM to 5PM, and will be in the "Red Experience" room at:

Novotel Ottawa
33 Nicholas Street
Ottawa
Ontario
K1N 9M7

Food and drink will be provided throughout the day, including breakfast from 8AM.

[http://maps.google.ca/maps?f=q&source=s_q&hl=en&geocode=&q=novotel+ottawa&aq=&sll=49.891235,-97.15369&sspn=36.237851,79.013672&ie=UTF8&hq=novotel+ottawa&hnear=&ll=45.421528,-75.683699&spn=0.036869,0.077162&z=14&iwloc=A&layer=c&cbll=45.425741,-75.689638&panoid=Z4FUGnkZkdHAOkIxyjjS9Q&cbp=12,25.83,,0,-0.6 View on Google Maps]

== Attendees ==

The following people have RSVPed to the meeting (in alphabetical order, by surname):

* Oleg Bartunov
* Josh Berkus (Secretary)
* Jeff Davis
* Andrew Dunstan
* Dimitri Fontaine
* Stephen Frost
* Peter Geoghegan
* Kevin Grittner
* Robert Haas
* Magnus Hagander
* Shigeru Hanada
* Hitoshi Harada
* KaiGai Kohei
* Tom Lane
* Noah Misch
* Bruce Momjian
* Dave Page (Chair)
* Simon Riggs
* Teodor Sigaev
* Greg Smith

== Proposed Agenda Items ==

Please list proposed agenda items here:

* Agree CommitFest schedule for 9.3 (Strawman from Simon)
** CF1 June 15, 2012 - 1 month
** CF2 Sep 15, 2012 - 1 month
** CF3 Nov 15, 2012 - 1 month
** CF4 Jan 15, 2013 - 2 months
* Priorities for 9.3 [All]
** Description: discuss what people are working on and what's likely to be in 9.3.
** Goals: set expectations and coordinate work schedules for 9.3.
* Queuing [Dimitri, Kevin]
** Description: efficient and transactional queuing is a very common need for application using databases, and could help implementing some internal features
** Goals: get an agreement that core is the right place where to solve that problem, and what parts of it we want in core exactly
* Materialized views [Kevin]
** Description: Declarative materialized views are a frequently requested feature, but means many things to many people. It's not likely that an initial implementation will address everything. We need a base set of functionality on which to build.
** Goals: Reach consensus on what a minimum feature set for commit would be.
* Partitioning and Segment Exclusion [Dimitri]
** Description: to solve partitioning, we need to agree on a global approach
** Goals: agreeing on SE as a basis for better partitioning, having a "GO" on working on SE
* The MERGE statement: Challenges and priorities [Peter G]
** Description: Implementing the MERGE statement for 9.3. It is envisaged specifically as an atomic "upsert" operation.
** Goals: To get buy-in on various aspects of the feature's development, and, ideally, to secure reviewer resources or other support. Because of the complexity of the feature, early interest from reviewers is preferable.
* Row-level Access Control and SELinux [KaiGai]
** Security label on user tables
** Dynamic expandable enum data types
** Enforcement of triggers by extension
* Enhancement of FDW at v9.3 [KaiGai]
** Writable foreign tables
** Stuffs to be pushed down (Join, Aggregate, Sort, ...)
** Inheritance of foreign/regular tables
** Constraint (PK/FK) & Trigger support.
* Type registry [Andrew]
** Provide for known OIDs for non-builtin types, and possibly for their IO functions too
** Would make it possible to write code in core or in extension X that handles a type defined in extension Y.
* Ending CommitFests in a timely fashion, especially the last one. Avoiding a crush of massive feature patches at the end of the cycle. Handling big patches that aren't quite ready yet. Getting more people to help with patch review. [Robert]
* What Developers Want [Josh]
** Description: a top-5 list of features and obstacles to developer adoption of PostgreSQL (with slides)
** Goal: to set priorities for some features aimed at application users
* In-Place Upgrades & Checksums [Greg Smith, Simon]
** Description: Revisit in-place upgrades of the page format, now that pg_upgrade is available and multiple checksum implementations needing it have been proposed.
** Goal: Nail down some incremental milestones for 9.3 development to aim at.
* Autonomous Transactions [Simon]
** Overview of idea, relationship to stored procedures
** Feedback, buy-in and/or alternatives
* Parallel Query [Bruce Momjian]
** Hope to get buy-in for what parallel operations we are hoping to add in upcoming releases
* Report from Clustering Meeting [Josh] (10 min)
** Description: to summarize the discussions of the cluster-hackers meeting from the previous day
** Goal: inter-team synchronization. Possibly, decisions requested on specific in-core features.
* Double Write Buffers [Simon]
** Is anyone committing to do that for 9.3?
* Summarise Commitments at End of Play [Simon]
** For roadmap and planning purposes, confirm who is doing what, assign interested reviewers at start
** Check gaps, identif priorities early on in cycle

== Agenda ==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter
|- style="font-style:italic;background-color:lightgray;"
|08:00
|Breakfast
|

|- style="font-style:italic;background-color:lightgray;"
|08:30 - 08:45
|Welcome and introductions
|Dave Page

|-
|08:45 - 09:10
|Goals for 9.3
|Josh Berkus

|-
|09:10 - 09:35
|Commitfest management
|Robert Haas

|-
|09:35 - 09:50
|9.3 commitfest schedule
|Simon Riggs

|-
|09:50 - 10:10
|Type registry
|Andrew Dunstan

|-
|10:10 - 10:30
|Access control and SELinux
|KaiGai Kohei

|- style="font-style:italic;background-color:lightgray;"
|10:30 - 10:45
|Coffee break
|

|-
|10:45 - 11:15
|Enhancement of FDWs in 9.3
|KaiGai Kohei

|-
|11:15 - 11:40
|Autonomous transactions
|Simon Riggs

|-
|11:40 - 12:05
|Partitioning and segment exclusion
|Dimitri Fontaine

|-
|12:05 - 12:30
|Queuing
|Dimitri Fontaine/Kevin Grittner

|- style="font-style:italic;background-color:lightgray;"
|12:30 - 13:30
|Lunch
|

|-
|13:30 - 14:00
|What developers want
|Josh Berkus

|-
|14:00 - 14:30
|The MERGE statement: Challenges and priorities
|Peter Geoghegan

|-
|14:30 - 15:00
|Materialised views
|Kevin Grittner

|- style="font-style:italic;background-color:lightgray;"
|15:00 - 15:15
|Tea break
|

|-
|15:15 - 15:45
|In place upgrades and checksums
|Simon Riggs/Greg Smith

|-
|15:45 - 16:15
|Parallel Query
|Bruce Momjian

|-
|16:15 - 16:25
|Report from the Clustering Meeting
|Josh Berkus

|-
|16:25 - 16:45
|Summarise commitments and identify priorities
|Simon Riggs

|- style="font-style:italic;background-color:lightgray;"
|16:45 - 17:00
|Any other business/group photo
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|17:00
|Finish
|
|}

==Minutes==

Lock dependency information

Kgrittn — Mon, 07 May 2012 17:31:47 GMT

Kgrittn: Use section headers to visually separate the two views.

{{SnippetInfo|Lock dependency info|lang=SQL|category=Performance}}
At times it is very usefull to see which locks depend uppon each other.

== Flat View of Blocking ==

All columns prefixed with ''waiting_'' hold information about the not granted lock, while the colums prefixed with ''other_'' hold information about other locks on the same relation respectively transactionid
<source lang="SQL">SELECT
waiting.locktype AS waiting_locktype,
waiting.relation::regclass AS waiting_table,
waiting_stm.current_query AS waiting_query,
waiting.mode AS waiting_mode,
waiting.pid AS waiting_pid,
other.locktype AS other_locktype,
other.relation::regclass AS other_table,
other_stm.current_query AS other_query,
other.mode AS other_mode,
other.pid AS other_pid,
other.granted AS other_granted
FROM
pg_catalog.pg_locks AS waiting
JOIN
pg_catalog.pg_stat_activity AS waiting_stm
ON (
waiting_stm.procpid = waiting.pid
)
JOIN
pg_catalog.pg_locks AS other
ON (
(
waiting."database" = other."database"
AND waiting.relation = other.relation
)
OR waiting.transactionid = other.transactionid
)
JOIN
pg_catalog.pg_stat_activity AS other_stm
ON (
other_stm.procpid = other.pid
)
WHERE
NOT waiting.granted
AND
waiting.pid <> other.pid
</source>

It would be useful to add extra columns indicating how long the waiting statement has been blocked.

== Recursive View of Blocking ==

<source lang="SQL">WITH RECURSIVE
c(requested, current) AS
( VALUES
('AccessShareLock'::text, 'AccessExclusiveLock'::text),
('RowShareLock'::text, 'ExclusiveLock'::text),
('RowShareLock'::text, 'AccessExclusiveLock'::text),
('RowExclusiveLock'::text, 'ShareLock'::text),
('RowExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('RowExclusiveLock'::text, 'ExclusiveLock'::text),
('RowExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ShareLock'::text, 'RowExclusiveLock'::text),
('ShareLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareLock'::text, 'ShareRowExclusiveLock'::text),
('ShareLock'::text, 'ExclusiveLock'::text),
('ShareLock'::text, 'AccessExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'RowExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ShareLock'::text),
('ShareRowExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ExclusiveLock'::text, 'RowShareLock'::text),
('ExclusiveLock'::text, 'RowExclusiveLock'::text),
('ExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ExclusiveLock'::text, 'ShareLock'::text),
('ExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ExclusiveLock'::text, 'ExclusiveLock'::text),
('ExclusiveLock'::text, 'AccessExclusiveLock'::text),
('AccessExclusiveLock'::text, 'AccessShareLock'::text),
('AccessExclusiveLock'::text, 'RowShareLock'::text),
('AccessExclusiveLock'::text, 'RowExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ShareLock'::text),
('AccessExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ExclusiveLock'::text),
('AccessExclusiveLock'::text, 'AccessExclusiveLock'::text)
),
l AS
(
SELECT
(locktype,database,relation::regclass::text,page,tuple,virtualxid,transactionid,classid,objid,objsubid) AS target,
virtualtransaction,
pid,
mode,
granted
FROM pg_catalog.pg_locks
),
t AS
(
SELECT
blocker.target AS blocker_target,
blocker.pid AS blocker_pid,
blocker.mode AS blocker_mode,
'1'::int AS depth,
blocked.target AS target,
blocked.pid AS pid,
blocked.mode AS mode,
blocker.pid::text || ',' || blocked.pid::text AS seq
FROM l blocker
JOIN l blocked
ON ( not blocked.granted
AND blocked.target IS NOT DISTINCT FROM blocker.target)
JOIN c ON (c.requested = blocked.mode AND c.current = blocker.mode)
UNION ALL
SELECT
blocker.target,
blocker.pid,
blocker.mode,
depth + 1,
blocked.target,
blocked.pid,
blocked.mode,
blocker.seq || ',' || blocked.pid::text
FROM t blocker
JOIN l blocked
ON ( not blocked.granted
AND blocked.target IS NOT DISTINCT FROM blocker.target)
JOIN c ON (c.requested = blocked.mode AND c.current = blocker.mode)
WHERE depth < 1000
)
SELECT target, blocker_pid, blocker_mode, depth, pid as blocked_pid, mode as blocked_mode, seq
FROM t ORDER BY seq;
</source>

PgCon 2012 Developer Meeting

Kgrittn — Mon, 07 May 2012 15:58:24 GMT

Kgrittn: /* Proposed Agenda Items */ Fill in description and goals for Materialized views.

A meeting of the most active PostgreSQL developers is being planned for Wednesday 16th May, 2012 near the University of Ottawa, prior to pgCon 2012. In order to keep the numbers manageable, this meeting is '''by invitation only'''. Unfortunately it is quite possible that we've overlooked important code developers during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that this year the attendee numbers have been cut to try to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.2 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies, unlike in previous years.

This is a PostgreSQL Community event. Room and refreshments/food sponsored by EnterpriseDB. Other companies sponsored attendance for their developers.

== Time & Location ==

The meeting will be from 9AM to 5PM, and will be in the "Red Experience" room at:

Novotel Ottawa
33 Nicholas Street
Ottawa
Ontario
K1N 9M7

Food and drink will be provided throughout the day, including breakfast from 8AM.

[http://maps.google.ca/maps?f=q&source=s_q&hl=en&geocode=&q=novotel+ottawa&aq=&sll=49.891235,-97.15369&sspn=36.237851,79.013672&ie=UTF8&hq=novotel+ottawa&hnear=&ll=45.421528,-75.683699&spn=0.036869,0.077162&z=14&iwloc=A&layer=c&cbll=45.425741,-75.689638&panoid=Z4FUGnkZkdHAOkIxyjjS9Q&cbp=12,25.83,,0,-0.6 View on Google Maps]

== Attendees ==

The following people have RSVPed to the meeting (in alphabetical order, by surname):

* Oleg Bartunov
* Josh Berkus (Secretary)
* Jeff Davis
* Andrew Dunstan
* Dimitri Fontaine
* Stephen Frost
* Peter Geoghegan
* Kevin Grittner
* Robert Haas
* Magnus Hagander
* Shigeru Hanada
* Hitoshi Harada
* KaiGai Kohei
* Tom Lane
* Noah Misch
* Bruce Momjian
* Dave Page (Chair)
* Simon Riggs
* Teodor Sigaev
* Greg Smith

== Proposed Agenda Items ==

Please list proposed agenda items here:

* Agree CommitFest schedule for 9.3 (Strawman from Simon)
** CF1 June 15, 2012 - 1 month
** CF2 Sep 15, 2012 - 1 month
** CF3 Nov 15, 2012 - 1 month
** CF4 Jan 15, 2013 - 2 months
* Priorities for 9.3 [All]
** Description: discuss what people are working on and what's likely to be in 9.3.
** Goals: set expectations and coordinate work schedules for 9.3.
* Queuing [Dimitri, Kevin]
** Description: efficient and transactional queuing is a very common need for application using databases, and could help implementing some internal features
** Goals: get an agreement that core is the right place where to solve that problem, and what parts of it we want in core exactly
* Materialized views [Kevin]
** Description: Declarative materialized views are a frequently requested feature, but means many things to many people. It's not likely that an initial implementation will address everything. We need a base set of functionality on which to build.
** Goals: Reach consensus on what a minimum feature set for commit would be.
* Partitioning and Segment Exclusion [Dimitri]
** Description: to solve partitioning, we need to agree on a global approach
** Goals: agreeing on SE as a basis for better partitioning, having a "GO" on working on SE
* The MERGE statement: Challenges and priorities [Peter G]
** Description: Implementing the MERGE statement for 9.3. It is envisaged specifically as an atomic "upsert" operation.
** Goals: To get buy-in on various aspects of the feature's development, and, ideally, to secure reviewer resources or other support. Because of the complexity of the feature, early interest from reviewers is preferable.
* Row-level Access Control and SELinux [KaiGai]
** Security label on user tables
** Dynamic expandable enum data types
** Enforcement of triggers by extension
* Enhancement of FDW at v9.3 [KaiGai]
** Writable foreign tables
** Stuffs to be pushed down (Join, Aggregate, Sort, ...)
** Inheritance of foreign/regular tables
** Constraint (PK/FK) & Trigger support.
* Type registry [Andrew]
** Provide for known OIDs for non-builtin types, and possibly for their IO functions too
** Would make it possible to write code in core or in extension X that handles a type defined in extension Y.
* Ending CommitFests in a timely fashion, especially the last one. Avoiding a crush of massive feature patches at the end of the cycle. Handling big patches that aren't quite ready yet. Getting more people to help with patch review. [Robert]
* What Developers Want [Josh]
** Description: a top-5 list of features and obstacles to developer adoption of PostgreSQL (with slides)
** Goal: to set priorities for some features aimed at application users
* In-Place Upgrades & Checksums [Greg Smith, Simon]
** Description: Revisit in-place upgrades of the page format, now that pg_upgrade is available and multiple checksum implementations needing it have been proposed.
** Goal: Nail down some incremental milestones for 9.3 development to aim at.
* Autonomous Transactions [Simon]
** Overview of idea, relationship to stored procedures
** Feedback, buy-in and/or alternatives
* Parallel Query [Bruce Momjian]
** Hope to get buy-in for what parallel operations we are hoping to add in upcoming releases
* Report from Clustering Meeting [Josh] (10 min)
** Description: to summarize the discussions of the cluster-hackers meeting from the previous day
** Goal: inter-team synchronization. Possibly, decisions requested on specific in-core features.
* Double Write Buffers [Simon]
** Is anyone committing to do that for 9.3?
* Summarise Commitments at End of Play [Simon]
** For roadmap and planning purposes, confirm who is doing what, assign interested reviewers at start
** Check gaps, identif priorities early on in cycle

== Agenda ==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter
|- style="font-style:italic;background-color:lightgray;"
|08:00
|Breakfast
|
|- style="font-style:italic;background-color:lightgray;"
|08:45 - 09:00
|Welcome and introductions
|Dave Page
|-
|- style="font-style:italic;background-color:lightgray;"
|10:30 - 10:45
|Coffee break
|
|- style="font-style:italic;background-color:lightgray;"
|12:30 - 13:30
|Lunch
|
|- style="font-style:italic;background-color:lightgray;"
|15:00 - 15:15
|Tea break
|
|- style="font-style:italic;background-color:lightgray;"
|16:45 - 17:00
|Any other business/group photo
|Dave Page
|- style="font-style:italic;background-color:lightgray;"
|17:00
|Finish
|
|}

==Minutes==

Lock dependency information

Kgrittn — Wed, 02 May 2012 20:59:07 GMT

Kgrittn: Eliminate duplicate target from final results, and rename some final result columns.

{{SnippetInfo|Lock dependency info|lang=SQL|category=Performance}}
At times it is very usefull to see which locks depend uppon each other.

All columns prefixed with ''waiting_'' hold information about the not granted lock, while the colums prefixed with ''other_'' hold information about other locks on the same relation respectively transactionid
<source lang="SQL">SELECT
waiting.locktype AS waiting_locktype,
waiting.relation::regclass AS waiting_table,
waiting_stm.current_query AS waiting_query,
waiting.mode AS waiting_mode,
waiting.pid AS waiting_pid,
other.locktype AS other_locktype,
other.relation::regclass AS other_table,
other_stm.current_query AS other_query,
other.mode AS other_mode,
other.pid AS other_pid,
other.granted AS other_granted
FROM
pg_catalog.pg_locks AS waiting
JOIN
pg_catalog.pg_stat_activity AS waiting_stm
ON (
waiting_stm.procpid = waiting.pid
)
JOIN
pg_catalog.pg_locks AS other
ON (
(
waiting."database" = other."database"
AND waiting.relation = other.relation
)
OR waiting.transactionid = other.transactionid
)
JOIN
pg_catalog.pg_stat_activity AS other_stm
ON (
other_stm.procpid = other.pid
)
WHERE
NOT waiting.granted
AND
waiting.pid <> other.pid
</source>

It would be useful to add extra columns indicating how long the waiting statement has been blocked.

Here's an attempt at a recursive version:

<source lang="SQL">WITH RECURSIVE
c(requested, current) AS
( VALUES
('AccessShareLock'::text, 'AccessExclusiveLock'::text),
('RowShareLock'::text, 'ExclusiveLock'::text),
('RowShareLock'::text, 'AccessExclusiveLock'::text),
('RowExclusiveLock'::text, 'ShareLock'::text),
('RowExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('RowExclusiveLock'::text, 'ExclusiveLock'::text),
('RowExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ShareLock'::text, 'RowExclusiveLock'::text),
('ShareLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareLock'::text, 'ShareRowExclusiveLock'::text),
('ShareLock'::text, 'ExclusiveLock'::text),
('ShareLock'::text, 'AccessExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'RowExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ShareLock'::text),
('ShareRowExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ExclusiveLock'::text, 'RowShareLock'::text),
('ExclusiveLock'::text, 'RowExclusiveLock'::text),
('ExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ExclusiveLock'::text, 'ShareLock'::text),
('ExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ExclusiveLock'::text, 'ExclusiveLock'::text),
('ExclusiveLock'::text, 'AccessExclusiveLock'::text),
('AccessExclusiveLock'::text, 'AccessShareLock'::text),
('AccessExclusiveLock'::text, 'RowShareLock'::text),
('AccessExclusiveLock'::text, 'RowExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ShareLock'::text),
('AccessExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ExclusiveLock'::text),
('AccessExclusiveLock'::text, 'AccessExclusiveLock'::text)
),
l AS
(
SELECT
(locktype,database,relation::regclass::text,page,tuple,virtualxid,transactionid,classid,objid,objsubid) AS target,
virtualtransaction,
pid,
mode,
granted
FROM pg_catalog.pg_locks
),
t AS
(
SELECT
blocker.target AS blocker_target,
blocker.pid AS blocker_pid,
blocker.mode AS blocker_mode,
'1'::int AS depth,
blocked.target AS target,
blocked.pid AS pid,
blocked.mode AS mode,
blocker.pid::text || ',' || blocked.pid::text AS seq
FROM l blocker
JOIN l blocked
ON ( not blocked.granted
AND blocked.target IS NOT DISTINCT FROM blocker.target)
JOIN c ON (c.requested = blocked.mode AND c.current = blocker.mode)
UNION ALL
SELECT
blocker.target,
blocker.pid,
blocker.mode,
depth + 1,
blocked.target,
blocked.pid,
blocked.mode,
blocker.seq || ',' || blocked.pid::text
FROM t blocker
JOIN l blocked
ON ( not blocked.granted
AND blocked.target IS NOT DISTINCT FROM blocker.target)
JOIN c ON (c.requested = blocked.mode AND c.current = blocker.mode)
WHERE depth < 1000
)
SELECT target, blocker_pid, blocker_mode, depth, pid as blocked_pid, mode as blocked_mode, seq
FROM t ORDER BY seq;
</source>

Lock dependency information

Kgrittn — Wed, 02 May 2012 20:53:08 GMT

Kgrittn: Start depth of recursive query at 1 instead of 0.

Lock dependency information

Kgrittn — Fri, 27 Apr 2012 20:14:31 GMT

Kgrittn: Add a first attempt at a recursive version of the query.

{{SnippetInfo|Lock dependency info|lang=SQL|category=Performance}}
At times it is very usefull to see which locks depend uppon each other.

All columns prefixed with ''waiting_'' hold information about the not granted lock, while the colums prefixed with ''other_'' hold information about other locks on the same relation respectively transactionid
<source lang="SQL">SELECT
waiting.locktype AS waiting_locktype,
waiting.relation::regclass AS waiting_table,
waiting_stm.current_query AS waiting_query,
waiting.mode AS waiting_mode,
waiting.pid AS waiting_pid,
other.locktype AS other_locktype,
other.relation::regclass AS other_table,
other_stm.current_query AS other_query,
other.mode AS other_mode,
other.pid AS other_pid,
other.granted AS other_granted
FROM
pg_catalog.pg_locks AS waiting
JOIN
pg_catalog.pg_stat_activity AS waiting_stm
ON (
waiting_stm.procpid = waiting.pid
)
JOIN
pg_catalog.pg_locks AS other
ON (
(
waiting."database" = other."database"
AND waiting.relation = other.relation
)
OR waiting.transactionid = other.transactionid
)
JOIN
pg_catalog.pg_stat_activity AS other_stm
ON (
other_stm.procpid = other.pid
)
WHERE
NOT waiting.granted
AND
waiting.pid <> other.pid
</source>

It would be useful to add extra columns indicating how long the waiting statement has been blocked.

Here's an attempt at a recursive version:

<source lang="SQL">WITH RECURSIVE
c(requested, current) AS
( VALUES
('AccessShareLock'::text, 'AccessExclusiveLock'::text),
('RowShareLock'::text, 'ExclusiveLock'::text),
('RowShareLock'::text, 'AccessExclusiveLock'::text),
('RowExclusiveLock'::text, 'ShareLock'::text),
('RowExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('RowExclusiveLock'::text, 'ExclusiveLock'::text),
('RowExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareLock'::text),
('ShareUpdateExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'ExclusiveLock'::text),
('ShareUpdateExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ShareLock'::text, 'RowExclusiveLock'::text),
('ShareLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareLock'::text, 'ShareRowExclusiveLock'::text),
('ShareLock'::text, 'ExclusiveLock'::text),
('ShareLock'::text, 'AccessExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'RowExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ShareLock'::text),
('ShareRowExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'ExclusiveLock'::text),
('ShareRowExclusiveLock'::text, 'AccessExclusiveLock'::text),
('ExclusiveLock'::text, 'RowShareLock'::text),
('ExclusiveLock'::text, 'RowExclusiveLock'::text),
('ExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('ExclusiveLock'::text, 'ShareLock'::text),
('ExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('ExclusiveLock'::text, 'ExclusiveLock'::text),
('ExclusiveLock'::text, 'AccessExclusiveLock'::text),
('AccessExclusiveLock'::text, 'AccessShareLock'::text),
('AccessExclusiveLock'::text, 'RowShareLock'::text),
('AccessExclusiveLock'::text, 'RowExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ShareUpdateExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ShareLock'::text),
('AccessExclusiveLock'::text, 'ShareRowExclusiveLock'::text),
('AccessExclusiveLock'::text, 'ExclusiveLock'::text),
('AccessExclusiveLock'::text, 'AccessExclusiveLock'::text)
),
l AS
(
SELECT
(locktype,database,relation::regclass::text,page,tuple,virtualxid,transactionid,classid,objid,objsubid) AS target,
virtualtransaction,
pid,
mode,
granted
FROM pg_catalog.pg_locks
),
t AS
(
SELECT
blocker.target AS blocker_target,
blocker.pid AS blocker_pid,
blocker.mode AS blocker_mode,
'0'::int AS depth,
blocked.target AS target,
blocked.pid AS pid,
blocked.mode AS mode,
blocker.pid::text || ',' || blocked.pid::text AS seq
FROM l blocker
JOIN l blocked
ON ( not blocked.granted
AND blocked.target IS NOT DISTINCT FROM blocker.target)
JOIN c ON (c.requested = blocked.mode AND c.current = blocker.mode)
UNION ALL
SELECT
blocker.target,
blocker.pid,
blocker.mode,
depth + 1,
blocked.target,
blocked.pid,
blocked.mode,
blocker.seq || ',' || blocked.pid::text
FROM t blocker
JOIN l blocked
ON ( not blocked.granted
AND blocked.target IS NOT DISTINCT FROM blocker.target)
JOIN c ON (c.requested = blocked.mode AND c.current = blocker.mode)
WHERE depth < 1000
)
SELECT * FROM t ORDER BY seq;
</source>

Guide to reporting problems

Kgrittn — Wed, 25 Apr 2012 17:29:28 GMT

Kgrittn: Replace "TL;DR:" section with (hopefully) friendlier text making the same point.

Please read all the way through this document, even though it is pretty long. It provides information which will help you to frame your question or state your problem in a way which will allow others to be more helpful. If any of the suggested information is missing, you may need to go back-and-forth with people while they try to gather enough information to help, or they may make guesses in the absence of actual data which could lead to off-target answers.

== Why were you sent this link? ==

Your question or post probably '''didn't include enough information for anyone to be able to help you''' or required too much guesswork to answer.

'''If you don't read and act the advice in this document, you will probably not get useful help. If you read this and act on it you will probably get better help, faster.'''

As a bonus, you'll often figure out your own question half way through writing an explanation of it to the mailing list.

== How to get it right (and get good help more quickly) ==

When writing requests for help, think about the reader and read your message from their point of view. What questions will they have to ask?

Remember, they can't see your screen. They don't know what you are trying to do. They can't tell what programs or OS you use.

'''The people reading your question only know what you tell them.''' The better the information you give, the better help they can offer.

Remember that the postgresql-general mailing list is populated by people helping out ''in their spare time''. Show that you respect the time they're spending to help you by following the advice given here before posting.

== Particular kinds of problem ==

'''Slow query?''' Read: [[SlowQueryQuestions|Guide to Posting Slow Query Questions]].

'''Installation problems?''' Read: [[Troubleshooting Installation]] and (on Windows) the [[Running & Installing PostgreSQL On Native Windows#Common installation errors|Windows installation FAQ]].

'''For (rare but nasty) suspected data/index corruption issues:''' [[Corruption|What to do if you suspect your database or indexes are corrupt]]

=== Things you need to mention in problem reports ===

To get a quick and helpful response, you must include at least the information shown in the list below. '''If you leave things out, your question may not be answered or you will be sent another link to this page and told to try again'''. Save yourself time: do it right the first time.

Please do not send screen shots or photographs of text. Make sure you copy and paste the text into the report email instead.

Make sure you include:

* '''A description of what you are trying to achieve and what results you expect'''.
** Describe ''in as much detail as possible'', step by step, including command lines, SQL output, etc.
* The '''EXACT PostgreSQL version''' you are running
** Run "SELECT version();" in <code>psql</code> or PgAdmin III and ''provide the full, exact output''. Paste it
* '''How you installed PostgreSQL'''
** Downloaded the EnterpriseDB One-click installer?
** From Linux distro package management? If so, what repository?
** Direct downloads of rpm/deb packages? From where?
** From BSD ports, MacPorts, etc?
** Downloaded and compiled the sources. If so, what options did you pass to <code>configure</code>? What compiler and version did you use?
** If you're having installer problems, make sure to include the installer log from the temporary folder. Windows users see [[Running & Installing PostgreSQL On Native Windows]].
* '''Changes made to the settings in the postgresql.conf file''': see [[Server Configuration]] for a quick way to list them all.
* '''Operating system and version'''
** Linux users:
*** Linux distro and version
*** Kernel details (run <code>uname -a</code> on the terminal)
** Windows users:
*** This means your Windows OS version, variant, and service pack. For example, "Windows XP Pro Service Pack 3". You can get this from the "winver" command - "Start -> Run -> winver.exe".
*** Whether you're running a 32-bit or 64-bit version of Windows
* For questions about any kind of error:
** '''What you were doing when the error happened / how to cause the error.'''
** '''The EXACT TEXT of the error message you're getting''' if there is one. Copy and paste the message to the email, do not send a screenshot.
* '''What program you're using to connect to PostgreSQL'''
** What version of the ODBC/JDBC/ADO/etc driver you're using, if any
** If you're using a connection pool, load balancer or application server, which one you're using and its version
* '''Is there anything remotely unusual in the PostgreSQL server logs?'''
** On Linux this depends a bit on distro, but you'll usually find them in <code>/var/log/postgresql/</code>.
** On Windows these are in your data directory. On a default PostgreSQL install that'll be in <code>C:\Program Files\PostgreSQL\8.4\data\pg_log</code> (assuming you're using 8.4)
** Windows users should also check the Event Viewer for service startup messages.

For errors or problems with queries:

* '''The EXACT text of the query you ran, if any'''
* '''The EXACT output of that query''' if it's short enough to be reasonable to post, otherwise a sample of it
* The SQL definition of any tables, views and user-defined functions your query uses, or psql <code>\d+</code> output for them.
* If at all possible, a '''self contained test case''' demonstrating your problem
* If you think the query result is wrong, what you think should've been produced instead and why
* For slow query problems, the information in the [[SlowQueryQuestions|Guide to Posting Slow Query Questions]] page.

Additionally, if your question relates to performance (query speed, memory use, etc) and/or data corruption, please include information about:

* CPU manufacturer and model, eg "AMD Athlon X2" or "Intel Core 2 Duo"
* Amount and size of RAM installed, eg "2GB RAM"
* Storage details (important for performance and corruption questions)
** Do you use a RAID controller? If so, what type of controller? eg "3Ware Escalade 8500-8"
*** Does it have a battery backed cache module?
*** Is write-back caching enabled?
** Do you use software RAID? If so, what software and what version? eg "Linux software RAID (md) 2.6.18-5-686 SMP mod_unload 686 REGPARM gcc-4.1".
*** In the case of Linux software RAID you can get the details from the "modinfo md_mod" command
** Is your PostgreSQL database on a SAN?
*** Who made it, what kind, etc? Provide what details you can.
** How many hard disks are connected to the system and what types are they? You need to say more than just "6 disks". At least give maker, otational speed and interface type, eg "6 15,000rpm Seagate SAS disks".
** How are your disks arranged for storage? Are you using RAID? If so, what RAID level(s)? What PostgreSQL data is on what disks / disk sets? What file system(s) are in use?
*** eg: "Two disks in RAID 1, with all PostgreSQL data and programs stored on one ext3 file system."
*** eg: "4 disks in RAID 5 holding the pg data directory on an ext3 file system. 2 disks in RAID 1 holding pg_clog, pg_xlog, the temporary tablespace, and the sort scratch space, also on ext3.".
*** eg: "Default Windows install of PostgreSQL"
** In case of corruption data reports:
*** Have you ''ever'' set <code>fsync=off</code> in the postgresql config file?
*** Have you had any unexpected power loss lately? Replaced a failed RAID disk? Had an operating system crash?
*** Have you run a file system check? (<code>chkdsk</code> / <code>fsck</code>)
*** Are there any error messages in the system logs? (unix/linux: <code>dmesg</code>, <code>/var/log/syslog</code> ; Windows: Event Viewer in Control Panel -> Administrative Tools )

== Things Not To Do ==

People who answer your questions on mailing lists aren't being paid to do so. They're doing it out of community spirit and a desire to help other PostgreSQL users, so that they can get help when they need it. As such, they have to ''want'' to help you; if you make yourself obnoxious, you won't get any assistance. Here's a number of bad practices which will make people more likely to ignore your request than answer it:

* '''"It's an Emergency!"''': community support is peer-to-peer, free, and at the helper's convenience. If it's a real emergency, get a support contract with a commercial support company. Most of them will do per-incident support without an existing contract.
* '''Cross-Posting''': do not post the same question to 2 or more mailing lists at the same time. Not only does this annoy the people you want help from, it's likely to get your e-mails spam-filtered out. If you post it to one list, and don't get a response in 2 days or more, then it's ok to post it to another list.
* '''User Questions on Hacker Lists''': pgsql-hackers, pgsql-committers, pgsql-testers, pgsql-www and pgsql-rrreviewers are all for people who work on the PostgreSQL database engine and our community infrastructure. Asking user questions on any of those is more liable to get you flamed than any answers. There are over 3 dozen user mailing lists, use them. Many code contributors and committers do read the -general, -bugs, etc lists, so you'll be heard if there's something wrong.
* '''Insist on Asking on the Wrong List''': sometimes you will post a question on one mailing list and will be told that you need to post it on a different list. Don't insist on pursuing the topic on the original list to everyone's irritation.
* '''Send Email Directly to a List Member Without Copying the List''': when someone responds on the list to try to help, any reply you make to that should normally be kept on the list. There are several reasons for this; among them are that the discussion may be useful to others who later run into the same problem, and that there may be other members of the community who can also provide help.
* '''Re-posting''': don't repost your question multiple times to the same list. If people haven't answered, they're busy, don't know the answer, or don't want to help you. If a week goes by without a response, you might "reply" to your own post, with an "Help? Anyone?", but more likely you should ask about what other list might be more appropriate for the question. If you re-post, sometimes discussion gets split across the threads under your different posts, confusing everybody and making it harder to help you.
* '''Comparisons with Other DBMSes''' are generally not helpful, unless your request is highly technical and is the result of serious comparison testing. We're generally not interested in replicating exactly how Oracle or MySQL do things; if we were, we'd work on those databases instead. Furthermore, many readers of the -general list may not be familiar with the database system you're talking about, so saying "it's like <blah>" doesn't help much. Explain what you're trying to do, not how you did it in some other database. (That said, if you notice an issue where Pg supports something, but not with quite the same syntax as another major database, it's worth mentioning that since sometimes the other syntax can be easily added for compatibility).
* '''"Postgres Is Broken!"''' as well as variations on "I'm going to abandon Postgres if you don't help me" will not usually get you help, and certainly don't get you help any faster. The general reaction you get will be: "Nobody is forcing you to use PostgreSQL, feel free to use MySQL/Oracle/CouchDB". Similarly, starting off your discussion by accusing PostgreSQL of having a bug because it doesn't work the way you expect is also not a good way to begin.
* '''Insist That Someone's Answer is Wrong (without testing)''': if you think you know better than they do, why are you asking for help? Usually what you actually mean is ''thanks for your suggestion, but you seem to have misunderstood my question. Perhaps I explained it poorly, let me try again''. Or, perhaps, ''That doesn't quite achieve what I'm trying to do, because it doesn't ''blah'' - is there anything that might?''. Think about how you say things.

[[Category:FAQ]] [[Category:Asking Questions]] [[Category:Administration]]

Todo

Kgrittn — Tue, 20 Mar 2012 16:56:11 GMT

Kgrittn: /* Locking */ Correct minor vertical whitespace anomaly.

<div style="margin: 1ex 1em; float: right;">
__TOC__
</div>

This list contains '''all known PostgreSQL bugs and feature requests'''. If you would like to work on an item, please read the [[Developer FAQ]] first. There is also a [[Development_information|development information page]].

* {{TodoPending}} - marks ordinary, incomplete items
* {{TodoEasy}} - marks items that are easier to implement
* {{TodoDone}} - marks changes that are done, and will appear in the PostgreSQL 9.2 release.

For help on editing this list, please see [[Talk:Todo]]. Please do not add items here without discussion on the mailing list.

<div style="padding: 1ex 4em;">
== Administration ==

{{TodoItem
|Allow administrators to cancel multi-statement idle transactions
|This allows locks to be released, but it is complex to report the cancellation back to the client.
* [http://archives.postgresql.org/pgsql-hackers/2008-12/msg01340.php <nowiki>Cancelling idle in transaction state</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-12/msg00441.php <nowiki>Re: Cancelling idle in transaction state</nowiki>]
}}

{{TodoItem
|Check for unreferenced table files created by transactions that were in-progress when the server terminated abruptly
* [http://archives.postgresql.org/pgsql-patches/2006-06/msg00096.php <nowiki>Removing unreferenced files</nowiki>]
}}

{{TodoItem
|Set proper permissions on non-system schemas during db creation
|Currently all schemas are owned by the super-user because they are copied from the template1 database. However, since all objects are inherited from the template database, it is not clear that setting schemas to the db owner is correct.}}

{{TodoItem
|Allow log_min_messages to be specified on a per-module basis
|This would allow administrators to see more detailed information from specific sections of the backend, e.g. checkpoints, autovacuum, etc. Another idea is to allow separate configuration files for each module, or allow arbitrary SET commands to be passed to them. See also [[Logging Brainstorm]].}}

{{TodoItem
|Simplify creation of partitioned tables
|This would allow creation of partitioned tables without requiring creation of triggers or rules for INSERT/UPDATE/DELETE, and constraints for rapid partition selection. Options could include range and hash partition selection. See also [[Table partitioning]]
}}

{{TodoItem
|Allow custom variables to appear in pg_settings()
* [http://archives.postgresql.org/pgsql-hackers/2008-06/msg00850.php <nowiki>Re: count(*) performance improvement ideas</nowiki>]
}}

{{TodoItem
|Have custom variables be transaction-safe
* {{MessageLink|4B577E9F.8000505@dunslane.net|Custom GUCs still a bit broken}}
}}

{{TodoItem
|Implement the SQL-standard mechanism whereby REVOKE ROLE revokes only the privilege granted by the invoking role, and not those granted by other roles
* [http://archives.postgresql.org/pgsql-bugs/2007-05/msg00010.php <nowiki>Re: Grantor name gets lost when grantor role dropped</nowiki>]
}}

{{TodoItem
|Prevent query cancel packets from being replayed by an attacker, especially when using SSL
* [http://archives.postgresql.org/pgsql-hackers/2008-08/msg00345.php <nowiki>Replay attack of query cancel</nowiki>]
}}

{{TodoItem
|Provide a way to query the log collector subprocess to determine the name of the currently active log file
* [http://archives.postgresql.org/pgsql-general/2008-11/msg00418.php <nowiki>Current log files when rotating?</nowiki>]
}}

{{TodoItem
|Allow simpler reporting of the unix domain socket directory and allow easier configuration of its default location
* http://archives.postgresql.org/pgsql-hackers/2010-10/msg01555.php
* http://archives.postgresql.org/pgsql-hackers/2011-10/msg01482.php
}}

{{TodoItem
|Allow custom daemons to be automatically stopped/started along with the postmaster
|This allows easier administration of daemons like user job schedulers or replication-related daemons.
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01701.php <nowiki>Re: scheduler in core</nowiki>]
}}

{{TodoItem
|Improve logging of prepared transactions recovered during startup
* [http://archives.postgresql.org/pgsql-hackers/2006-11/msg00092.php <nowiki>"recovering prepared transaction" after server restart message</nowiki>]
}}

{{TodoItem
|Consider using POSIX shared memory to avoid System V shared memory kernel limits
* [http://archives.postgresql.org/message-id/4DFA2673.3010009@enterprisedb.com <nowiki>POSIX shared memory patch status</nowiki>]
}}

{{TodoItemDone
|Address problem where superusers are assumed to be members of all groups
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00337.php
}}

=== Configuration files ===
{{TodoSubsection}}

{{TodoItemEasy
|Change pg_ident.conf parsing to be the same as pg_hba.conf
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg02204.php
}}

{{TodoItem
|Allow postgresql.conf file values to be changed via an SQL API, perhaps using SET GLOBAL
* http://archives.postgresql.org/pgsql-hackers/2010-10/msg00764.php
}}

{{TodoItem
|Consider normalizing fractions in postgresql.conf, perhaps using '%'
* [http://archives.postgresql.org/pgsql-hackers/2007-06/msg00550.php <nowiki>Fractions in GUC variables</nowiki>]
}}

{{TodoItem
|Allow Kerberos to disable stripping of realms so we can check the username@realm against multiple realms
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg00009.php <nowiki>krb_match_realm patch</nowiki>]
}}

{{TodoItem
|Improve LDAP authentication configuration options
* [http://archives.postgresql.org/pgsql-hackers/2008-04/msg01745.php <nowiki>Proposed Patch - LDAPS support for servers on port 636 w/o TLS</nowiki>]
}}

{{TodoItem
|Add external tool to auto-tune some postgresql.conf parameters
* [http://archives.postgresql.org/pgsql-hackers/2008-06/msg00000.php <nowiki>Re: Overhauling GUCS</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-11/msg00033.php <nowiki>Simple postgresql.conf wizard</nowiki>]
}}

{{TodoItem
|Add 'hostgss' pg_hba.conf option to allow GSS link-level encryption
* [http://archives.postgresql.org/pgsql-hackers/2008-07/msg01454.php <nowiki>Re: Plans for 8.4</nowiki>]
}}

{{TodoItem
|Process pg_hba.conf keywords as case-insensitive
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg00432.php <nowiki>More robust pg_hba.conf parsing/error logging</nowiki>]
}}

{{TodoItem
|Create utility to compute accurate random_page_cost value
* http://archives.postgresql.org/pgsql-performance/2011-04/msg00162.php
* http://archives.postgresql.org/pgsql-performance/2011-04/msg00362.php
}}

{{TodoItem
|Allow configuration files to be independently validated
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01831.php
* http://archives.postgresql.org/message-id/12666.1310774573@sss.pgh.pa.us
}}

{{TodoItem
|Allow postgresql.conf settings to be accepted by backends even if some settings are invalid for those backends
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00330.php
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg00375.php
}}

{{TodoItem
|Allow all backends to receive postgresql.conf setting changes at the same time
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00330.php
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg00375.php
}}

{{TodoEndSubsection}}

=== Tablespaces ===
{{TodoSubsection}}

{{TodoItem
|Allow a database in tablespace t1 with tables created in tablespace t2 to be used as a template for a new database created with default tablespace t2
|Currently all objects in the default database tablespace must have default tablespace specifications. This is because new databases are created by copying directories. If you mix default tablespace tables and tablespace-specified tables in the same directory, creating a new database from such a mixed directory would create a new database with tables that had incorrect explicit tablespaces. To fix this would require modifying pg_class in the newly copied database, which we don't currently do.}}

{{TodoItem
|Allow reporting of which objects are in which tablespaces
|This item is difficult because a tablespace can contain objects from multiple databases. There is a server-side function that returns the databases which use a specific tablespace, so this requires a tool that will call that function and connect to each database to find the objects in each database for that tablespace.}}

{{TodoItem
|Allow WAL replay of CREATE TABLESPACE to work when the directory structure on the recovery computer is different from the original}}

{{TodoItem
|Allow per-tablespace quotas}}

{{TodoItem
|Allow tablespaces on RAM-based partitions for unlogged tables
* http://archives.postgresql.org/pgsql-advocacy/2011-05/msg00033.php
}}

{{TodoItem
|Allow toast tables to be moved to a different tablespace
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg00980.php
* {{messageLink|CAFEQCbH756DyyAPQ1ykh3+b+kE1-EhWRww1WO_x5v38C-uLnUg@mail.gmail.com|patch : Allow toast tables to be moved to a different tablespace}} (issues remain)
* [http://archives.postgresql.org/message-id/CAFEQCbEq07OopgE5xFYv2Q3eMq45hRSJkjCBO+kvpJq9NEVhow@mail.gmail.com Allow toast tables to be moved to a different tablespace]
}}

{{TodoEndSubsection}}

=== Statistics Collector ===
{{TodoSubsection}}

{{TodoItem
|Allow statistics last vacuum/analyze execution times to be displayed without requiring track_counts to be enabled
* [http://archives.postgresql.org/pgsql-docs/2007-04/msg00028.php <nowiki>row-level stats and last analyze time</nowiki>]
}}

{{TodoItem
|Clear table counters on TRUNCATE
* [http://archives.postgresql.org/pgsql-hackers/2008-04/msg00169.php <nowiki>Small TRUNCATE glitch</nowiki>]
}}

{{TodoEndSubsection}}

=== SSL ===
{{TodoSubsection}}

{{TodoItem
|Allow SSL authentication/encryption over unix domain sockets
* [http://archives.postgresql.org/pgsql-hackers/2007-12/msg00924.php <nowiki>Re: Spoofing as the postmaster</nowiki>]
}}

{{TodoItem
|Allow SSL key file permission checks to be optionally disabled when sharing SSL keys with other applications
* [http://archives.postgresql.org/pgsql-bugs/2007-12/msg00069.php <nowiki>BUG #3809: SSL "unsafe" private key permissions bug</nowiki>]
}}

{{TodoItem
|Allow SSL CRL files to be re-read during configuration file reload, rather than requiring a server restart
|Unlike SSL CRT files, CRL (Certificate Revocation List) files are updated frequently
* [http://archives.postgresql.org/pgsql-general/2008-12/msg00832.php <nowiki>Automatic CRL reload</nowiki>]
Alternatively or additionally supporting OCSP (online certificate security protocol) would provide real-time revocation discovery without reloading
}}

{{TodoItem
| Allow automatic selection of SSL client certificates from a certificate store
* [http://archives.postgresql.org/pgsql-hackers/2009-05/msg00406.php <nowiki>Allow multiple certificates or keys in the postgresql.crt/.key files</nowiki>]
}}

{{TodoItem
| Send the full certificate server chain to the client
* [http://archives.postgresql.org/pgsql-bugs/2009-12/msg00145.php BUG #5245: Full Server Certificate Chain Not Sent to client]
}}

{{TodoEndSubsection}}

=== Point-In-Time Recovery (PITR) ===
{{TodoSubsection}}

{{TodoItemEasy
|Create dump tool for write-ahead logs for use in determining transaction id for point-in-time recovery
|This is useful for checking PITR recovery.}}

{{TodoItem
|Allow archive_mode to be changed without server restart?
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg01655.php <nowiki>Enabling archive_mode without restart</nowiki>]
}}

{{TodoItem
|Consider avoiding WAL switching via archive_timeout if there has been no database activity
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01469.php <nowiki>archive_timeout behavior for no activity</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00395.php <nowiki>Re: archive_timeout behavior for no activity</nowiki>]
}}

{{TodoEndSubsection}}

=== Standby server mode ===
{{TodoSubsection}}

{{TodoItem
| Allow pg_xlogfile_name() to be used in recovery mode
* [http://archives.postgresql.org/message-id/3f0b79eb1001190135vd9f62f1sa7868abc1ea61d12@mail.gmail.com <nowiki>Streaming replication and pg_xlogfile_name()</nowiki>]
}}

{{TodoItem
| Prevent variables inherited from the server environment from begin used for making streaming replication connections.
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01011.php <nowiki>Re: Parameter name standby_mode</nowiki>]
}}

{{TodoItemDone
| Allow hot file system backups on standby servers
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg01727.php
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01490.php
}}

{{TodoItem
| Change walsender so that it applies per-role settings
* http://archives.postgresql.org/pgsql-hackers/2010-09/msg00642.php
}}

{{TodoItemDone
| Add more control over waiting for synchronous commit
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01611.php
}}

{{TodoItem
| Restructure configuration parameters for standby mode
* http://archives.postgresql.org/pgsql-hackers/2010-09/msg01820.php
}}

{{TodoItem
| Allow time-delayed application of logs on the standby
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00992.php
}}
{{TodoItem
| Add -X parameter to pg_basebackup to specify a different directory for px_xlog, like initdb
}}

{{TodoEndSubsection}}

== Data Types ==

{{TodoItem
|Fix data types where equality comparison is not intuitive, e.g. box
* http://archives.postgresql.org/pgsql-hackers/2011-10/msg01643.php
}}

{{TodoItem
|Add support for public SYNONYMs
* [http://archives.postgresql.org/pgsql-hackers/2006-03/msg00519.php <nowiki>Proposal for SYNONYMS</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg02043.php
* http://archives.postgresql.org/pgsql-general/2010-12/msg00139.php
}}

{{TodoItem
|Add support for SQL-standard GENERATED/IDENTITY columns
* [http://archives.postgresql.org/pgsql-hackers/2006-07/msg00543.php <nowiki>Re: Three weeks left until feature freeze</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2006-08/msg00038.php <nowiki>GENERATED ... AS IDENTITY, Was: Re: Feature Freeze</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-05/msg00344.php <nowiki>Behavior of GENERATED columns per SQL2003</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2007-05/msg00076.php <nowiki>Re: [HACKERS] Behavior of GENERATED columns per SQL2003</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00604.php <nowiki>IDENTITY/GENERATED patch</nowiki>]
}}

{{TodoItem
|Consider placing all sequences in a single table, or create a system view
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00008.php <nowiki>Re: newbie: renaming sequences task</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2012-02/msg00258.php Removing special case OID generation]
}}

{{TodoItem
|Consider a special data type for regular expressions
* [http://archives.postgresql.org/pgsql-hackers/2007-08/msg01067.php <nowiki>Why is there a tsquery data type?</nowiki>]
}}

{{TodoItem
|Reduce BIT data type overhead using short varlena headers
* [http://archives.postgresql.org/pgsql-general/2007-12/msg00273.php <nowiki>storage size of "bit" data type..</nowiki>]
}}

{{TodoItem
|Allow renaming and deleting enumerated values from an existing enumerated data type
}}

{{TodoItem
|Support scoped IPv6 addresses in the inet type
* [http://archives.postgresql.org/pgsql-bugs/2007-05/msg00111.php <nowiki>strange problem with ip6</nowiki>]
}}

{{TodoItemDone
|Add a JSON (JavaScript Object Notation) data type
|This would behave similar to the XML data type, which is stored as text, but allows element lookup and conversion functions.
* [http://archives.postgresql.org/pgsql-hackers/2009-12/msg01494.php <nowiki>PATCH: Add hstore_to_json()</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg00001.php <nowiki>Re: PATCH: Add hstore_to_json()</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-03/msg01092.php <nowiki>Proposal: Add JSON support</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-04/msg00057.php <nowiki>Re: Proposal: Add JSON support</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg00481.php
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01694.php
* http://archives.postgresql.org/pgsql-hackers/2011-12/msg00219.php
}}

{{TodoItem
|Considering improving performance of computing CHAR() value lengths
* [http://archives.postgresql.org/pgsql-hackers/2009-06/msg00900.php <nowiki>char() overhead on read-only workloads not so insignifcant as the docs claim it is...</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01787.php <nowiki>Re: [PATCH] backend: compare word-at-a-time in bcTruelen</nowiki>]
}}

{{TodoItem
|Add overlaps geometric operators that ignore point overlaps
* http://archives.postgresql.org/pgsql-hackers/2010-03/msg00861.php
}}

{{TodoItem
| Add IMMUTABLE column attribute
* http://archives.postgresql.org/pgsql-hackers/2011-11/msg00623.php
}}

=== Domains ===
{{TodoSubsection}}

{{TodoItem
|Allow functions defined as casts to domains to be called during casting
* [http://archives.postgresql.org/pgsql-hackers/2006-05/msg00072.php <nowiki>bug? non working casts for domain</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2006-09/msg01681.php <nowiki>TODO: Fix CREATE CAST on DOMAINs</nowiki>]
}}

{{TodoItem
|Allow values to be cast to domain types
* [http://archives.postgresql.org/pgsql-hackers/2003-06/msg01206.php <nowiki>Domain casting still doesn't work right</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-08/msg00289.php <nowiki>domain casting?</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg00812.php
}}

{{TodoItem
|Make domains work better with polymorphic functions
* [http://archives.postgresql.org/message-id/4887.1228700773@sss.pgh.pa.us Polymorphic types vs. domains]
* [http://archives.postgresql.org/message-id/15535.1238774571@sss.pgh.pa.us some difficulties with fixing it]
}}

{{TodoEndSubsection}}

=== Dates and Times ===
{{TodoSubsection}}

{{TodoItem
|Allow infinite intervals just like infinite timestamps
* http://archives.postgresql.org/pgsql-hackers/2011-11/msg00076.php
}}

{{TodoItem
|Determine how to represent date/time field extraction on infinite timestamps
* [http://archives.postgresql.org/message-id/CA+mi_8bda-Fnev9iXeUbnqhVaCWzbYhHkWoxPQfBca9eDPpRMw@mail.gmail.com extract(epoch from infinity) is not 0]
* [http://archives.postgresql.org/message-id/CADAkt-icuESH16uLOCXbR-dKpcvwtUJE4JWXnkdAjAAwP6j12g@mail.gmail.com converting between infinity timestamp and float8]
}}

{{TodoItem
|Allow TIMESTAMP WITH TIME ZONE to store the original timezone information, either zone name or offset from UTC
|If the TIMESTAMP value is stored with a time zone name, interval computations should adjust based on the time zone rules.
* [http://archives.postgresql.org/pgsql-hackers/2004-10/msg00705.php <nowiki>timestamp with time zone a la sql99</nowiki>]
}}

{{TodoItem
|Have timestamp subtraction not call justify_hours()?
* [http://archives.postgresql.org/pgsql-sql/2006-10/msg00059.php <nowiki>timestamp subtraction (was Re: formatting intervals with to_char)</nowiki>]
}}

{{TodoItem
|Improve TIMESTAMP WITH TIME ZONE subtraction to be DST-aware
|Currently subtracting one date from another that crosses a daylight savings time adjustment can return '1 day 1 hour', but adding that back to the first date returns a time one hour in the future. This is caused by the adjustment of '25 hours' to '1 day 1 hour', and '1 day' is the same time the next day, even if daylight savings adjustments are involved.}}

{{TodoItem
|Fix interval display to support values exceeding 2^31 hours}}

{{TodoItem
|Add overflow checking to timestamp and interval arithmetic}}

{{TodoItem
|Add function to allow the creation of timestamps using parameters
* http://archives.postgresql.org/pgsql-performance/2010-06/msg00232.php
}}

{{TodoEndSubsection}}

=== Arrays ===
{{TodoSubsection}}

{{TodoItem
|Add support for arrays of domains
* [http://archives.postgresql.org/pgsql-patches/2007-05/msg00114.php <nowiki>Re: updated WIP: arrays of composites</nowiki>]
}}

{{TodoItem
|Allow single-byte header storage for array elements}}

{{TodoItem
|Add function to detect if an array is empty
* [http://archives.postgresql.org/pgsql-hackers/2008-11/msg00475.php <nowiki>Re: array_length()</nowiki>]
}}

{{TodoItem
|Improve handling of empty arrays
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg01033.php <nowiki>So what's an "empty" array anyway?</nowiki>]
}}

{{TodoItem
|Improve handling of NULLs in arrays
* [http://archives.postgresql.org/pgsql-bugs/2008-11/msg00009.php <nowiki>BUG #4509: array_cat's null behaviour is inconsistent</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg01040.php
}}

{{TodoEndSubsection}}

=== Binary Data ===
{{TodoSubsection}}

{{TodoItem
|Improve vacuum of large objects, like contrib/vacuumlo?}}

{{TodoItem
|Auto-delete large objects when referencing row is deleted
|contrib/lo offers this functionality.}}

{{TodoItem
|Allow read/write into TOAST values like large objects
|Writing might require the TOAST column to be stored EXTERNAL.
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg00049.php
}}

{{TodoItem
|Add API for 64-bit large object access
* [http://archives.postgresql.org/pgsql-hackers/2005-09/msg00781.php <nowiki>64-bit API for large objects</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-09/msg01790.php
}}

{{TodoEndSubsection}}

=== MONEY Data Type ===
{{TodoSubsection}}

{{TodoItem
|Add locale-aware MONEY type, and support multiple currencies
* [http://archives.postgresql.org/pgsql-general/2005-08/msg01432.php <nowiki>A real currency type</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg01181.php <nowiki>Money type todos?</nowiki>]
}}

{{TodoItem
|MONEY dumps in a locale-specific format making it difficult to restore to a system with a different locale}}

{{TodoEndSubsection}}

=== Text Search ===
{{TodoSubsection}}

{{TodoItem
|Allow dictionaries to change the token that is passed on to later dictionaries
* [http://archives.postgresql.org/pgsql-patches/2007-11/msg00081.php <nowiki>a tsearch2 (8.2.4) dictionary that only filters out stopwords</nowiki>]
}}

{{TodoItem
|Consider a function-based API for '@@' searches
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg00511.php <nowiki>Simplifying Text Search</nowiki>]
}}

{{TodoItem
|Improve text search error messages
* [http://archives.postgresql.org/pgsql-hackers/2007-10/msg00966.php <nowiki>Poorly designed tsearch NOTICEs</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg01146.php <nowiki>Re: Poorly designed tsearch NOTICEs</nowiki>]
}}

{{TodoItem
|Consider changing error to warning for strings larger than one megabyte
* [http://archives.postgresql.org/pgsql-bugs/2008-02/msg00190.php <nowiki>BUG #3975: tsearch2 index should not bomb out of 1Mb limit</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-03/msg00062.php <nowiki>Re: [BUGS] BUG #3975: tsearch2 index should not bomb out of 1Mb limit</nowiki>]
}}

{{TodoItem
|tsearch and tsdicts regression tests fail in Turkish locale on glibc
* [http://archives.postgresql.org/message-id/49749645.5070801@gmx.net tsearch with Turkish locale]
}}

{{TodoItem
|tsquery negator operator treated as part of lexeme
* [http://archives.postgresql.org/pgsql-bugs/2009-06/msg00346.php BUG #4887: inclusion operator (@>) on tsqeries behaves not conforming to documentation]
}}

{{TodoItem
|Improve handling of dash and plus signs in email address user names, and perhaps improve URL parsing
* http://archives.postgresql.org/pgsql-hackers/2010-10/msg00772.php
* [http://archives.postgresql.org/message-id/E1Ri8il-0008Ct-9p@wrigleys.postgresql.org tsearch does not recognize all valid emails]
}}

{{TodoItem
|Improve default parser, to more easily allow adding new tokens
* http://archives.postgresql.org/message-id/23485.1297727826@sss.pgh.pa.us
}}

{{TodoItem
|Add additional support functions
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg00319.php
}}

{{TodoEndSubsection}}

=== XML ===
{{TodoSubsection}}

{{TodoItem
|Allow XML arrays to be cast to other data types
* [http://archives.postgresql.org/pgsql-hackers/2007-09/msg00981.php <nowiki>proposal casting from XML[] to int[], numeric[], text[]</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-10/msg00231.php <nowiki>Re: proposal casting from XML[] to int[], numeric[], text[]</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg00471.php <nowiki>Re: proposal casting from XML[] to int[], numeric[], text[]</nowiki>]
}}

{{TodoItem
|Add XML Schema validation and xmlvalidate functions (SQL:2008)}}

{{TodoItem
|Add xmlvalidatedtd variant to support validating against a DTD?}}

{{TodoItem
|Relax-NG validation; libxml2 supports this already}}

{{TodoItem
|Allow reliable XML operation non-UTF8 server encodings (xpath(), in particular, is known to not work)
* [http://archives.postgresql.org/pgsql-bugs/2009-01/msg00135.php <nowiki>BUG #4622: xpath only work in utf-8 server encoding</nowiki>]
* http://archives.postgresql.org/message-id/4110.1238973350@sss.pgh.pa.us}}

{{TodoItem
|Add functions from SQL:2006: XMLDOCUMENT, XMLCAST, XMLTEXT}}

{{TodoItem
|Add XMLNAMESPACES support in XMLELEMENT and elsewhere}}

{{TodoItem
|Move XSLT from contrib/xml2 to a more reasonable location
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg00539.php
}}

{{TodoItem
|Report errors returned by the XSLT library
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg00562.php
}}

{{TodoItem
|Improve the XSLT parameter passing API
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg00416.php
}}

{{TodoItem
|XML Canonical: Convert XML documents to canonical form to compare them. libxml2 has support for this.}}

{{TodoItem
|Add pretty-printed XML output option
|Parse a document and serialize it back in some indented form. libxml2 might support this.}}

{{TodoItem
|Add XMLQUERY (from the SQL/XML standard)}}

{{TodoItem
|Allow XML sthredding
|In some cases shredding could be better option (if there is no need to keep XML docs entirely, e.g. if we have already developed tools that understand only relational data. This would be a separate module that implements annotated schema decomposition technique, similar to DB2 and SQL Server functionality.}}

{{TodoItem
|Fix Nested or repeated xpath() that apparently mess up namespaces [http://archives.postgresql.org/pgsql-bugs/2008-03/msg00097.php] [http://archives.postgresql.org/pgsql-bugs/2008-03/msg00144.php] [http://archives.postgresql.org/pgsql-general/2008-03/msg00295.php] [http://archives.postgresql.org/pgsql-bugs/2008-07/msg00054.php] [http://archives.postgresql.org/message-id/004f01c90e91$138e9d10$3aabd730$@anstett@iaas.uni-stuttgart.de]}}

{{TodoItem
|XPath: Adding the <x> at the root causes problems [http://archives.postgresql.org/pgsql-bugs/2008-05/msg00184.php] [http://archives.postgresql.org/pgsql-bugs/2008-07/msg00054.php] [http://archives.postgresql.org/pgsql-general/2008-07/msg00613.php]}}

{{TodoItem
|xpath_table needs to be implemented/implementable to get rid of contrib/xml2 [http://archives.postgresql.org/pgsql-general/2008-05/msg00823.php]}}

{{TodoItem
|xpath_table is pretty broken anyway [http://archives.postgresql.org/pgsql-hackers/2010-02/msg02424.php]}}

{{TodoItem
|better handling of XPath data types [http://archives.postgresql.org/pgsql-hackers/2008-06/msg00616.php] [http://archives.postgresql.org/message-id/004a01c90e90$4b986d90$e2c948b0$@anstett@iaas.uni-stuttgart.de]}}

{{TodoItem
|Improve handling of PIs and DTDs in xmlconcat() [http://archives.postgresql.org/message-id/200904211211.n3LCB09p008988@wwwmaster.postgresql.org]}}

{{TodoItem
|Restructure XML and /contrib/xml2 functionality
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg02314.php
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg00017.php
}}

{{TodoEndSubsection}}

== Functions ==

{{TodoItem
|Allow INET subnet comparisons using non-constants to be indexed}}

{{TodoItem
|Add an INET overlaps operator, for use by exclusion constraints
* http://archives.postgresql.org/pgsql-hackers/2010-03/msg00845.php
}}

{{TodoItem
|Enforce typmod for function inputs, function results and parameters for spi_prepare'd statements called from PLs
* [http://archives.postgresql.org/pgsql-hackers/2007-01/msg01403.php <nowiki>Re: BUG #2917: spi_prepare doesn't accept typename aliases</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-11/msg01160.php <nowiki>RFC for adding typmods to functions</nowiki>]
}}

{{TodoItem
|Fix IS OF so it matches the ISO specification, and add documentation
* [http://archives.postgresql.org/pgsql-patches/2003-08/msg00060.php <nowiki>Re: [HACKERS] IS OF</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-02/msg00060.php <nowiki>ToDo: add documentation for operator IS OF</nowiki>]
}}

{{TodoItem
|Implement Boyer-Moore searching in LIKE queries
* {{messageLink|27645.1220635769@sss.pgh.pa.us|TODO item: Implement Boyer-Moore searching (First time hacker)}}
}}

{{TodoItem
|Prevent malicious functions from being executed with the permissions of unsuspecting users
|Index functions are safe, so VACUUM and ANALYZE are safe too. Triggers, CHECK and DEFAULT expressions, and rules are still vulnerable.
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00268.php <nowiki>Some notes about the index-functions security vulnerability</nowiki>]
}}

{{TodoItem
|Reduce memory usage of aggregates in set returning functions
* [http://archives.postgresql.org/pgsql-performance/2008-01/msg00031.php <nowiki>Re: Performance of aggregates over set-returning functions</nowiki>]
}}

{{TodoItem
|Fix /contrib/ltree operator
* [http://archives.postgresql.org/pgsql-bugs/2007-11/msg00044.php <nowiki>BUG #3720: wrong results at using ltree</nowiki>]
}}

{{TodoItem
|Fix /contrib/btree_gist's implementation of inet indexing
* [http://archives.postgresql.org/pgsql-bugs/2010-10/msg00099.php <nowiki>BUG #5705: btree_gist: Index on inet changes query result</nowiki>]
}}

{{TodoItem
|<nowiki>Fix inconsistent precedence of =, >, and < compared to <>, >=, and <=</nowiki>
* [http://archives.postgresql.org/pgsql-bugs/2007-12/msg00145.php <nowiki>BUG #3822: Nonstandard precedence for comparison operators</nowiki>]
}}

{{TodoItem
|Fix regular expression bug when using complex back-references
* [http://archives.postgresql.org/pgsql-bugs/2007-10/msg00000.php <nowiki>BUG #3645: regular expression back references seem broken</nowiki>]
}}

{{TodoItem
|Have /contrib/dblink reuse unnamed connections
* [http://archives.postgresql.org/pgsql-hackers/2007-10/msg00895.php <nowiki>dblink un-named connection doesn't get re-used</nowiki>]
}}

{{TodoItem
|Improve formatting of pg_get_viewdef() output
* [http://archives.postgresql.org/pgsql-hackers/2009-01/msg01648.php <nowiki>pg_get_viewdef formattiing</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-08/msg01885.php <nowiki>Re: pretty print viewdefs</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2011-12/msg00906.php reprise: pretty print viewdefs]
}}

{{TodoItem
|Add function to dump pg_depend information cleanly
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg00226.php <nowiki>Elementary dependency look-up</nowiki>]
}}

{{TodoItemDone
|Improve relation size functions such as pg_relation_size() to avoid producing an error when called against a no longer visible relation
* [http://archives.postgresql.org/message-id/28488.1286461610@sss.pgh.pa.us pg_relation_size / could not open relation with OID #]
}}

=== Character Formatting ===

{{TodoSubsection}}
{{TodoItem
|Allow to_date() and to_timestamp() to accept localized month names}}

{{TodoItem
|Add missing parameter handling in to_char()
* [http://archives.postgresql.org/pgsql-hackers/2005-12/msg00948.php <nowiki>Re: to_char and i18n</nowiki>]
}}

{{TodoItem
|Throw an error from to_char() instead of printing a string of "#" when a number doesn't fit in the desired output format.
* discussed in [http://archives.postgresql.org/message-id/37ed240d0907290836w42187222n18664dfcbcb445b1@mail.gmail.com "to_char, support for EEEE format"]
}}

{{TodoItem
|Allow to_char() on interval values to accumulate the highest unit requested
|2= Some special format flag would be required to request such accumulation. Such functionality could also be added to EXTRACT. Prevent accumulation that crosses the month/day boundary because of the uneven number of days in a month.
* to_char(INTERVAL '1 hour 5 minutes', 'MI') => 65
* to_char(INTERVAL '43 hours 20 minutes', 'MI' ) => 2600
* to_char(INTERVAL '43 hours 20 minutes', 'WK:DD:HR:MI') => 0:1:19:20
* to_char(INTERVAL '3 years 5 months','MM') => 41
}}

{{TodoItem
|Fix to_number() handling for values not matching the format string
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg01447.php <nowiki>Re: numeric_to_number() function skipping some digits</nowiki>]
}}

{{TodoEndSubsection}}

== Multi-Language Support ==

{{TodoItem
|Add NCHAR (as distinguished from ordinary varchar),}}

{{TodoItem
|Add a cares-about-collation column to pg_proc, so that unresolved-collation errors can be thrown at parse time
* [http://archives.postgresql.org/pgsql-hackers/2011-03/msg01520.php <nowiki>Open issues for collations</nowiki>]
}}

{{TodoItem
|Integrate collations with text search configurations
* [http://archives.postgresql.org/message-id/28887.1303579034@sss.pgh.pa.us <nowiki>Some TODO items for collations</nowiki>]
}}

{{TodoItem
|Integrate collations with to_char() and related functions
* [http://archives.postgresql.org/message-id/28887.1303579034@sss.pgh.pa.us <nowiki>Some TODO items for collations</nowiki>]
}}

{{TodoItem
|Support collation-sensitive equality and hashing functions
* [http://archives.postgresql.org/pgsql-hackers/2011-06/msg00472.php <nowiki> contrib/citext versus collations</nowiki>]
}}

{{TodoItem
|Add a LOCALE option to CREATE DATABASE, as a shorthand
* [http://archives.postgresql.org/pgsql-hackers/2009-04/msg00119.php <nowiki> Re: 8.4 open items list</nowiki>]
}}

{{TodoItem
|Support multiple simultaneous character sets, per SQL:2008}}

{{TodoItem
|Improve UTF8 combined character handling?}}

{{TodoItem
|Add octet_length_server() and octet_length_client()}}

{{TodoItem
|Make octet_length_client() the same as octet_length()?}}

{{TodoItem
|Fix problems with wrong runtime encoding conversion for NLS message files}}

{{TodoItem
|Add URL to more complete multi-byte regression tests
* [http://archives.postgresql.org/pgsql-hackers/2005-07/msg00272.php <nowiki>Multi-byte and client side character encoding tests for copy command..</nowiki>]
}}

{{TodoItem
|Fix contrib/fuzzystrmatch to work with multibyte encodings
* [http://archives.postgresql.org/pgsql-bugs/2009-04/msg00047.php <nowiki> soundex function returns UTF-16 characters</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-04/msg00138.php <nowiki> dmetaphone woes</nowiki>]
}}

{{TodoItem
|Change memory allocation for multi-byte functions so memory is allocated inside conversion functions
|Currently we preallocate memory based on worst-case usage.}}

{{TodoItem
|Add ability to use case-insensitive regular expressions on multi-byte characters
|Currently it works for UTF-8, but not other multi-byte encodings
* [http://archives.postgresql.org/pgsql-hackers/2008-12/msg00433.php <nowiki>Regexps vs. locale</nowiki>]
* {{MessageLink|20091201210024.B1393753FB7@cvs.postgresql.org|A partial solution for UTF-8}}
}}

{{TodoItem
|Improve encoding of connection startup messages sent to the client
|Currently some authentication error messages are sent in the server encoding
* [http://archives.postgresql.org/pgsql-general/2008-12/msg00801.php <nowiki>encoding of PostgreSQL messages</nowiki>]
* [http://archives.postgresql.org/pgsql-general/2009-01/msg00005.php <nowiki>Re: encoding of PostgreSQL messages</nowiki>]
}}

{{TodoItem
|More sensible support for Unicode combining characters, normal forms
* http://archives.postgresql.org/message-id/200904141532.44618.peter_e@gmx.net
}}

== Views / Rules ==

{{TodoItem
|Automatically create rules on views so they are updateable, per SQL:2008
|We can only auto-create rules for simple views. For more complex cases users will still have to write rules manually.
* [http://archives.postgresql.org/pgsql-hackers/2006-03/msg00586.php <nowiki>Proposal for updatable views</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2006-08/msg00255.php <nowiki>Updatable views</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-01/msg01746.php <nowiki>Re: [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle</nowiki>]
* http://wiki.postgresql.org/wiki/Updatable_views
}}

{{TodoItem
|Add the functionality of the WITH CHECK OPTION clause to CREATE VIEW}}

{{TodoItem
|Allow VIEW/RULE recompilation when the underlying tables change
|This is both difficult and controversial.
* [http://archives.postgresql.org/pgsql-hackers/2009-12/msg01723.php Re: About "Allow VIEW/RULE recompilation when the underlying tables change"]
* [http://archives.postgresql.org/pgsql-hackers/2009-12/msg01724.php Re: About "Allow VIEW/RULE recompilation when the underlying tables change"]
* [http://archives.postgresql.org/message-id/CACk=U9NFSzWrEba8G5dZ=TZLy3_hx3QXGyCcKVWT=4iA1FjMuA@mail.gmail.com VIEW still referring to old name of field]
}}

{{TodoItem
|Make it possible to use RETURNING together with conditional DO INSTEAD rules, such as for partitioning setups
* [http://archives.postgresql.org/pgsql-hackers/2007-09/msg00577.php <nowiki>RETURNING and DO INSTEAD ... Intentional or not?</nowiki>]
}}

{{TodoItem
|Add the ability to automatically create materialized views
|Right now materialized views require the user to create triggers on the main table to keep the summary table current. SQL syntax should be able to manage the triggers and summary table automatically. A more sophisticated implementation would automatically retrieve from the summary table when the main table is referenced, if possible. See [[Materialized Views]] for implementation details
* [http://archives.postgresql.org/pgsql-hackers/2010-04/msg00479.php <nowiki>GSoC - proposal - Materialized Views in PostgreSQL</nowiki>]
}}

{{TodoItem
|Improve ability to modify views via ALTER TABLE
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00691.php <nowiki>Re: idea: storing view source in system catalogs</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-07/msg01410.php <nowiki>modifying views</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-08/msg00300.php <nowiki>Re: patch: Add columns via CREATE OR REPLACE VIEW</nowiki>]
}}

{{TodoItemDone
|Prevent low-cost functions from seeing unauthorized view rows
* [http://archives.postgresql.org/pgsql-hackers/2009-10/msg01346.php <nowiki>Using views for row-level access control is leaky</nowiki>]
}}

== SQL Commands ==

{{TodoItem
|Add CORRESPONDING BY to UNION/INTERSECT/EXCEPT
* [http://dissipatedheat.com/2011/11/10/how-not-to-write-a-patch-for-postgresql/ How not to write this patch.]
}}

{{TodoItem
|Improve type determination of unknown (NULL or quoted literal) result columns for UNION/INTERSECT/EXCEPT
* [http://archives.postgresql.org/message-id/9799.1302719551@sss.pgh.pa.us <nowiki>UNION construct type cast gives poor error message</nowiki>]
}}

{{TodoItem
|Add ROLLUP, CUBE, GROUPING SETS options to GROUP BY
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg00838.php <nowiki>WIP: grouping sets support</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-05/msg00466.php <nowiki>Implementation of GROUPING SETS (T431: Extended grouping capabilities)</nowiki>]
}}

{{TodoItem
|Allow prepared transactions with temporary tables created and dropped in the same transaction, and when an ON COMMIT DELETE ROWS temporary table is accessed
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00047.php <nowiki>Re: "could not open relation 1663/16384/16584: No such file or directory" in a specific combination of transactions with temp tables</nowiki>]
* [http://archives.postgresql.org/message-id/492543D5.9050904@enterprisedb.com A suggestion on how to implement this]
}}

{{TodoItem
|Add a GUC variable to warn about non-standard SQL usage in queries}}

{{TodoItem
|Add SQL-standard MERGE/REPLACE/UPSERT command
|MERGE is typically used to merge two tables. REPLACE or UPSERT command does UPDATE, or on failure, INSERT. See [[SQL MERGE]] for notes on the implementation details.
}}

{{TodoItem
|Add NOVICE output level for helpful messages
|For example, have it warn about unjoined tables. This could also control automatic sequence/index creation messages.
}}

{{TodoItem
|Allow NOTIFY in rules involving conditionals}}

{{TodoItem
|Allow EXPLAIN to identify tables that were skipped because of constraint_exclusion
}}

{{TodoItem
|Simplify dropping roles that have objects in several databases}}

{{TodoItem
|Allow the count returned by SELECT, etc to be represented as an int64 to allow a higher range of values}}

{{TodoItem
|Add support for WITH RECURSIVE ... CYCLE
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg00291.php <nowiki>WITH RECURSIVE ... CYCLE in vanilla SQL: issues with arrays of rows</nowiki>]}}

{{TodoItem
|Add DEFAULT .. AS OWNER so permission checks are done as the table owner
|This would be useful for SERIAL nextval() calls and CHECK constraints.}}

{{TodoItem
|Allow DISTINCT to work in multiple-argument aggregate calls}}

{{TodoItem
|Add comments on system tables/columns using the information in catalogs.sgml
|Ideally the information would be pulled from the SGML file automatically.}}

{{TodoItem
|Prevent the specification of conflicting transaction read/write options
* [http://archives.postgresql.org/pgsql-hackers/2009-01/msg00684.php <nowiki>Re: SET TRANSACTION and SQL Standard</nowiki>]
}}

{{TodoItem
|Support LATERAL subqueries
|Lateral subqueries can reference columns of tables defined outside the subquery at the same level, i.e. ''laterally''.
For example, a LATERAL subquery in a FROM clause could reference tables defined in the same FROM clause.
Currently only the columns of tables defined ''above'' subqueries are recognized.
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg00292.php <nowiki>LATERAL</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-10/msg00991.php <nowiki>Re: LATERAL</nowiki>]
* [http://archives.postgresql.org/message-id/4F5AA202.9020906@gmail.com lateral function as a subquery - WIP patch]
}}

{{TodoItem
|Prevent temporary tables created with ON COMMIT DELETE ROWS from repeatedly truncating the table on every commit if the table is already empty
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg00842.php
* http://archives.postgresql.org/pgsql-performance/2010-03/msg00392.php
* http://archives.postgresql.org/pgsql-performance/2010-04/msg00046.php
}}

{{TodoItem
|Allow DELETE and UPDATE to be used with LIMIT and ORDER BY
* http://archives.postgresql.org/pgadmin-hackers/2010-04/msg00078.php
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg01997.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00021.php
}}

{{TodoItemDone
|Improve caching of prepared query plans
}}

{{TodoItem
|Allow PREPARE of cursors}}

{{TodoItem
|Have DISCARD PLANS discard plans cached by functions
|DISCARD all should do the same.
* http://archives.postgresql.org/pgsql-hackers/2011-01/msg00431.php
}}

{{TodoItem
|Avoid multiple-evaluation of BETWEEN and IN arguments containing volatile expressions
* http://archives.postgresql.org/message-id/4D95B605.2020709@enterprisedb.com
}}

{{TodoItem
|Fix nested CASE-WHEN constructs
* http://archives.postgresql.org/message-id/4DDCEEB8.50602@enterprisedb.com
}}

=== CREATE ===
{{TodoSubsection}}

{{TodoItem
|Allow CREATE TABLE AS to determine column lengths for complex expressions like SELECT col1 || col2}}

{{TodoItem
|Have WITH CONSTRAINTS also create constraint indexes
* [http://archives.postgresql.org/pgsql-patches/2007-04/msg00149.php <nowiki>Re: CREATE TABLE LIKE INCLUDING INDEXES support</nowiki>]
}}

{{TodoItem
|Move NOT NULL constraint information to pg_constraint
|Currently NOT NULL constraints are stored in pg_attribute without any designation of their origins, e.g. primary keys. One manifest problem is that dropping a PRIMARY KEY constraint does not remove the NOT NULL constraint designation. Another issue is that we should probably force NOT NULL to be propagated from parent tables to children, just as CHECK constraints are. (But then does dropping PRIMARY KEY affect children?)
* http://archives.postgresql.org/message-id/19768.1238680878@sss.pgh.pa.us
* http://archives.postgresql.org/message-id/200909181005.n8IA5Ris061239@wwwmaster.postgresql.org
* http://archives.postgresql.org/pgsql-hackers/2011-07/msg01223.php
}}

{{TodoItem
|Prevent concurrent CREATE TABLE from sometimes returning a cryptic error message
* [http://archives.postgresql.org/pgsql-bugs/2007-10/msg00169.php <nowiki>BUG #3692: Conflicting create table statements throw unexpected error</nowiki>]
}}

{{TodoItem
|Add CREATE SCHEMA ... LIKE that copies a schema}}

{{TodoItem
|Fix CREATE OR REPLACE FUNCTION to not leave objects depending on the function in inconsistent state
* [http://archives.postgresql.org/pgsql-general/2008-08/msg00985.php indexes on functions and create or replace function]
}}

{{TodoItem
|Allow temporary tables to exist as empty by default in all sessions
* [http://archives.postgresql.org/pgsql-hackers/2007-07/msg00006.php <nowiki>what is difference between LOCAL and GLOBAL TEMP TABLES in PostgreSQL</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-04/msg01329.php <nowiki>idea: global temp tables</nowiki>]
* [http://archives.postgresql.org//pgsql-hackers/2009-05/msg00016.php <nowiki>Re: idea: global temp tables</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-04/msg01098.php <nowiki>global temporary tables</nowiki>]
}}

{{TodoItem
|Allow the creation of "distinct" types
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg01647.php <nowiki>Distinct types</nowiki>]
}}

{{TodoItem
|Consider analyzing temporary tables when they are first used in a query
|Autovacuum cannot analyze or vacuum temporary tables.
* [http://archives.postgresql.org/pgsql-hackers/2010-04/msg00416.php <nowiki>autovacuum and temp tables support</nowiki>]
}}

{{TodoItem
|Allow an unlogged table to be changed to logged
* http://archives.postgresql.org/pgsql-hackers/2011-01/msg00315.php
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00437.php
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg00323.php
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg00237.php
}}

{{TodoEndSubsection}}

=== UPDATE ===
{{TodoSubsection}}

{{TodoItem
|<nowiki>Allow UPDATE tab SET ROW (col, ...) = (SELECT...)</nowiki>
* [http://archives.postgresql.org/pgsql-hackers/2006-07/msg01308.php <nowiki>Re: [PATCHES] extension for sql update</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg00865.php <nowiki>UPDATE using sub selects</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2007-04/msg00315.php <nowiki>UPDATE using sub selects</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-03/msg00237.php <nowiki>Re: UPDATE using sub selects</nowiki>]
}}

{{TodoItem
|Research self-referential UPDATEs that see inconsistent row versions in read-committed mode
* [http://archives.postgresql.org/pgsql-hackers/2007-05/msg00507.php <nowiki>Concurrently updating an updatable view</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-06/msg00016.php <nowiki>Re: Do we need a TODO? (was Re: Concurrently updating anupdatable view)</nowiki>]
}}

{{TodoItem
|Improve performance of EvalPlanQual mechanism that rechecks already-updated rows
|This is related to the previous item, which questions whether it even has the right semantics
* [http://archives.postgresql.org/pgsql-bugs/2008-09/msg00045.php <nowiki>BUG #4401: concurrent updates to a table blocks one update indefinitely</nowiki>]
* [http://archives.postgresql.org/pgsql-bugs/2009-07/msg00302.php <nowiki>BUG #4945: Parallel update(s) gone wild</nowiki>]
}}

{{TodoEndSubsection}}

=== ALTER ===
{{TodoSubsection}}

{{TodoItem
|Have ALTER TABLE RENAME of a SERIAL column rename the sequence
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00008.php <nowiki>Re: newbie: renaming sequences task</nowiki>]
* [http://archives.postgresql.org/message-id/CADLWmXUV4LbLhMZL8rYMhCy72aZZLB5BSARPQVgoX0BrxA0FFg@mail.gmail.com renaming implicit sequences]
}}

{{TodoItem
|Have ALTER SEQUENCE RENAME rename the sequence name stored in the sequence table
* [http://archives.postgresql.org/pgsql-bugs/2007-09/msg00092.php <nowiki>BUG #3619: Renaming sequence does not update its 'sequence_name' field</nowiki>]
* [http://archives.postgresql.org/pgsql-bugs/2007-10/msg00007.php <nowiki>Re: BUG #3619: Renaming sequence does not update its 'sequence_name' field</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00008.php <nowiki>Re: newbie: renaming sequences task</nowiki>]
}}

{{TodoItemDone
|Allow ALTER TABLE ... ALTER CONSTRAINT ... RENAME or ALTER TABLE RENAME CONSTRAINT
* [http://archives.postgresql.org/pgsql-patches/2006-02/msg00168.php <nowiki>ALTER CONSTRAINT RENAME patch reverted</nowiki>]
}}

{{TodoItem
|Add ALTER DOMAIN to modify the underlying data type}}

{{TodoItem
|Allow ALTER TABLESPACE to move the tablespace to different directories}}

{{TodoItem
|Allow moving system tables to other tablespaces, where possible
|Currently non-global system tables must be in the default database tablespace. Global system tables can never be moved.}}

{{TodoItem
|Have ALTER INDEX update the name of a constraint using that index}}

{{TodoItem
|Allow column display reordering by recording a display, storage, and permanent id for every column?
* [http://archives.postgresql.org/pgsql-hackers/2006-12/msg00782.php <nowiki>Re: column ordering, was Re: [PATCHES] Enums patch v2</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-11/msg01029.php <nowiki>Column reordering in pg_dump</nowiki>]
* http://archives.postgresql.org/message-id/1324412114-sup-9608@alvh.no-ip.org
}}

{{TodoItem
|Allow deactivating (and reactivating) indexes via ALTER TABLE
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg01191.php
}}

{{TodoItem
|Add ALTER OPERATOR ... RENAME
|needs to consider effects of changing operator precedence
* [http://archives.postgresql.org/message-id/1322948781.26266.9.camel@vanquo.pezone.net Missing rename support]
}}

{{TodoItem
|Add ALTER TABLE ... RENAME RULE
* [http://archives.postgresql.org/message-id/1322948781.26266.9.camel@vanquo.pezone.net Missing rename support]
}}

{{TodoEndSubsection}}

=== CLUSTER ===
{{TodoSubsection}}

{{TodoItem
|Automatically maintain clustering on a table
|This might require some background daemon to maintain clustering during periods of low usage. It might also require tables to be only partially filled for easier reorganization. Another idea would be to create a merged heap/index data file so an index lookup would automatically access the heap data too. A third idea would be to store heap rows in hashed groups, perhaps using a user-supplied hash function.
* [http://archives.postgresql.org/pgsql-performance/2004-08/msg00350.php <nowiki>Equivalent praxis to CLUSTERED INDEX?</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00155.php <nowiki>Re: Grouped Index Tuples</nowiki>]
* http://community.enterprisedb.com/git/
* [http://archives.postgresql.org/pgsql-performance/2009-10/msg00346.php <nowiki>Re: maintain_cluster_order_v5.patch</nowiki>]
}}

{{TodoEndSubsection}}

=== COPY ===
{{TodoSubsection}}

{{TodoItem
|Allow COPY to report error lines and continue
|This requires the use of a savepoint before each COPY line is processed, with ROLLBACK on COPY failure.
* [http://archives.postgresql.org/pgsql-hackers/2007-12/msg00572.php <nowiki>Re: VLDB Features</nowiki>]
}}

{{TodoItem
|Allow COPY FROM to create index entries in bulk
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00811.php <nowiki>Batch update of indexes on data loading</nowiki>]
}}

{{TodoItem
|Allow COPY in CSV mode to control whether a quoted zero-length string is treated as NULL
|Currently this is always treated as a zero-length string, which generates an error when loading into an integer column
* [http://archives.postgresql.org/pgsql-hackers/2007-07/msg00905.php <nowiki>Re: [PATCHES] allow CSV quote in NULL</nowiki>]
}}

{{TodoItem
|Improve COPY performance
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00954.php <nowiki>Re: 8.3 / 8.2.6 restore comparison</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg01882.php
}}

{{TodoItem
|Allow COPY to report errors sooner
* [http://archives.postgresql.org/pgsql-hackers/2008-04/msg01169.php <nowiki>Timely reporting of COPY errors</nowiki>]
}}

{{TodoItem
|Allow COPY to handle other number formats
|E.g. the German notation. Best would be something like WITH DECIMAL ','.
}}

{{TodoItem
|Allow a stalled COPY to exit if the backend is terminated
* [http://archives.postgresql.org/pgsql-bugs/2009-04/msg00067.php <nowiki>Re: possible bug not in open items</nowiki>]
}}

{{TodoEndSubsection}}

=== GRANT/REVOKE ===
{{TodoSubsection}}

{{TodoItem
|Allow SERIAL sequences to inherit permissions from the base table?}}

{{TodoItem
|Allow dropping of a role that has connection rights
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00736.php <nowiki>DROP ROLE dependency tracking ...</nowiki>]
}}
{{TodoEndSubsection}}

=== DECLARE CURSOR ===
{{TodoSubsection}}

{{TodoItem
|Prevent DROP TABLE from dropping a table referenced by its own open cursor?}}

{{TodoItem
|Provide some guarantees about the behavior of cursors that invoke volatile functions
* [http://archives.postgresql.org/message-id/20997.1244563664@sss.pgh.pa.us Re: Cursor with hold emits the same row more than once across commits in 8.3.7]
}}

{{TodoEndSubsection}}

=== INSERT ===
{{TodoSubsection}}

{{TodoItem
|Allow INSERT/UPDATE of the system-generated oid value for a row}}

{{TodoItem
|In rules, allow VALUES() to contain a mixture of 'old' and 'new' references}}

{{TodoEndSubsection}}

=== SHOW/SET ===
{{TodoSubsection}}

{{TodoItem
|Add SET PERFORMANCE_TIPS option to suggest INDEX, VACUUM, VACUUM ANALYZE, and CLUSTER}}

{{TodoItem
|Rationalize the discrepancy between settings that use values in bytes and SHOW that returns the object count
* [http://archives.postgresql.org/pgsql-docs/2008-07/msg00007.php <nowiki>Re: [ADMIN] shared_buffers and shmmax</nowiki>]
}}

{{TodoEndSubsection}}

=== ANALYZE ===
{{TodoSubsection}}

{{TodoItem
|Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage}}

{{TodoItem
|Have EXPLAIN ANALYZE report rows as floating-point numbers
* [http://archives.postgresql.org/pgsql-hackers/2009-05/msg01363.php <nowiki>explain analyze rows=%.0f</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-06/msg00108.php <nowiki>Re: explain analyze rows=%.0f</nowiki>]
}}

{{TodoItem
|Improve how ANALYZE computes in-doubt tuples
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg00771.php <nowiki>VACUUM/ANALYZE counting of in-doubt tuples</nowiki>]
}}

{{TodoEndSubsection}}

=== Window Functions ===
See {{messageLink|357.1230492361@sss.pgh.pa.us|TODO items for window functions}}.
{{TodoSubsection}}
{{TodoItem
|Support creation of user-defined window functions
|We have the ability to create new window functions written in C. Is it
worth the effort to create an API that would let them be written in PL/pgsql, etc?}}

{{TodoItem
|Implement full support for window framing clauses
|In addition to done clauses described in the [http://developer.postgresql.org/pgdocs/postgres/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS latest doc], these clauses are not implemented yet.
* RANGE BETWEEN ... PRECEDING/FOLLOWING
* EXCLUDE
}}

{{TodoItem
|Investigate tuplestore performance issues
|The tuplestore_in_memory() thing is just a band-aid, we ought to try to solve it properly. tuplestore_advance seems like a weak spot as well.
* [http://archives.postgresql.org/pgsql-hackers/2008-12/msg00152.php <nowiki>tuplestore potential performance problem</nowiki>]
}}

{{TodoItem|Do we really need so much duplicated code between Agg and WindowAgg?}}

{{TodoItem
|Teach planner to evaluate multiple windows in the optimal order
|Currently windows are always evaluated in the query-specified order.
* http://archives.postgresql.org/message-id/3CDAD71E9D70417290FCF66F0178D1E1@amd64
}}

{{TodoItem
|Implement DISTINCT clause in window aggregates
|Some proprietary RDBMSs have implemented it already, so it helps with porting from those.}}

{{TodoEndSubsection}}

== Integrity Constraints ==
=== Keys ===

{{TodoSubsection}}

{{TodoItem
|Improve deferrable unique constraints for cases with many conflicts
|The current implementation fires a trigger for each potentially conflicting row. This might not scale well for an update that changes many key values at once.
}}

{{TodoEndSubsection}}

=== Referential Integrity ===
{{TodoSubsection}}

{{TodoItem
|Add MATCH PARTIAL referential integrity}}

{{TodoItem
|Change foreign key constraint for array -> element to mean element in array?
* [http://archives.postgresql.org/pgsql-hackers/2010-10/msg01814.php <nowiki>foreign keys for array/period contains relationships</nowiki>]
}}

{{TodoItem
|Fix problem when cascading referential triggers make changes on cascaded tables, seeing the tables in an intermediate state
* [http://archives.postgresql.org/pgsql-hackers/2005-09/msg00174.php <nowiki>Re: [PATCHES] Work-in-progress referential action trigger timing</nowiki>]
}}

{{TodoItem
|Optimize referential integrity checks
* [http://archives.postgresql.org/pgsql-performance/2005-10/msg00458.php <nowiki>Re: Effects of cascading references in foreign keys</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-04/msg00744.php <nowiki>Can't ri_KeysEqual() consider two nulls as equal?</nowiki>]
}}

{{TodoEndSubsection}}

=== Check Constraints ===
{{TodoSubsection}}

{{TodoItem
|Run check constraints only when affected columns are changed
* http://archives.postgresql.org/message-id/1326055327.15293.13.camel@vanquo.pezone.net
}}

{{TodoEndSubsection}}

== Server-Side Languages ==

{{TodoItem
|Add support for polymorphic arguments and return types to languages other than PL/PgSQL}}

{{TodoItem
|Add support for OUT and INOUT parameters to languages other than PL/PgSQL}}

{{TodoItem
|Add more fine-grained specification of functions taking arbitrary data types
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg00367.php <nowiki>RfD: more powerful "any" types</nowiki>]
}}

{{TodoItem
|Implement stored procedures
|This might involve the control of transaction state and the return of multiple result sets
* [http://archives.postgresql.org/pgsql-general/2008-10/msg00454.php <nowiki>PL/pgSQL stored procedure returning multiple result sets (SELECTs)?</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-10/msg01375.php <nowiki>Proposal: real procedures again (8.4)</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-09/msg00542.php
* [http://archives.postgresql.org/pgsql-hackers/2011-04/msg01149.php <nowiki>Gathering specs and discussion on feature (post 9.1)</nowiki>]
}}

{{TodoItem
|Allow holdable cursors in SPI}}

{{TodoItemEasy
|Add SPI_gettypmod() to return a field's typemod from a TupleDesc
* http://archives.postgresql.org/pgsql-hackers/2005-11/msg00250.php
}}

=== SQL-Language Functions ===
{{TodoSubsection}}

{{TodoItemDone
|Allow SQL-language functions to reference parameters by parameter name
|Currently SQL-language functions can only refer to dollar parameters, e.g. $1
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01479.php
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01519.php
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00221.php
}}

{{TodoItem
|Rethink query plan caching and timing of parse analysis within SQL-language functions
|They should work more like plpgsql functions do ...
* [http://archives.postgresql.org/pgsql-bugs/2011-05/msg00078.php <nowiki>Re: BUG #6019: invalid cached plan on inherited table</nowiki>]
}}

{{TodoEndSubsection}}

=== PL/pgSQL ===
{{TodoSubsection}}

{{TodoItem
|Allow handling of %TYPE arrays, e.g. tab.col%TYPE[]}}

{{TodoItem
|<nowiki>Allow listing of record column names, and access to record columns via variables, e.g. columns := r.(*), tval2 := r.(colname)</nowiki>
* [http://archives.postgresql.org/pgsql-patches/2005-07/msg00458.php <nowiki>Re: PL/PGSQL: Dynamic Record Introspection</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2006-05/msg00302.php <nowiki>Re: PL/PGSQL: Dynamic Record Introspection</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2006-06/msg00031.php <nowiki>Re: PL/PGSQL: Dynamic Record Introspection</nowiki>]
}}

{{TodoItem
|Allow row and record variables to be set to NULL constants, and allow NULL tests on such variables
|Because a row is not scalar, do not allow assignment from NULL-valued scalars.
* [http://archives.postgresql.org/pgsql-hackers/2006-10/msg00070.php <nowiki>NULL and plpgsql rows</nowiki>]
}}

{{TodoItem
|Consider keeping separate cached copies when search_path changes
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg01009.php <nowiki>pl/pgsql Plan Invalidation and search_path</nowiki>]
}}

{{TodoItem
|Improve handling of NULL row values vs. NULL rows
* [http://archives.postgresql.org/pgsql-hackers/2008-09/msg01758.php <nowiki>Null row vs. row of nulls in plpgsql</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-10/msg01973.php
}}

{{TodoItem
|Improve PERFORM handling of WITH queries or document limitation
* http://archives.postgresql.org/pgsql-bugs/2011-03/msg00309.php
}}

{{TodoEndSubsection}}

=== PL/Perl ===
{{TodoSubsection}}

{{TodoItem
|Allow regex operations in plperl using UTF8 characters in non-UTF8 encoded databases}}

{{TodoEndSubsection}}

=== PL/Python ===
{{TodoSubsection}}

{{TodoItem
|Develop a trusted variant of PL/Python.}}

{{TodoItem
|Create a new restricted execution class that will allow passing function arguments in as locals. Passing them as globals means functions cannot be called recursively.
* [http://archives.postgresql.org/pgsql-hackers/2011-02/msg01468.php <nowiki>Re: pl/python do not delete function arguments</nowiki>]
}}

{{TodoItem
|Add a DB-API compliant interface on top of the SPI interface
* http://petereisentraut.blogspot.com/2011/11/plpydbapi-db-api-for-plpython.html
}}

{{TodoItem
|For functions returning a setof record with a composite type, cache the I/O functions for the composite type
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg02007.php
}}

{{TodoItem
|Fix loss of information during conversion of numeric type to Python float}}

{{TodoEndSubsection}}

=== PL/Tcl ===
{{TodoSubsection}}

{{TodoItem
|Add table function support}}

{{TodoItem
|Check encoding validity of values passed back to Postgres in function returns, trigger tuple changes, and SPI calls.}}

{{TodoEndSubsection}}

== Clients ==

{{TodoItem
|Add a function like pg_get_indexdef() that report more detailed index information
* [http://archives.postgresql.org/pgsql-bugs/2007-12/msg00166.php <nowiki>BUG #3829: Wrong index reporting from pgAdmin III (v1.8.0 rev 6766-6767)</nowiki>]
}}

{{TodoItem
|Split out pg_resetxlog output into pre- and post-sections
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg02040.php
}}

=== pg_ctl ===
{{TodoSubsection}}

{{TodoItemDone
|Allow pg_ctl to work properly with configuration files located outside the PGDATA directory
|pg_ctl can not read the pid file because it isn't located in the config directory but in the PGDATA directory. The solution is to allow pg_ctl to read and understand postgresql.conf to find the data_directory value.
* [http://archives.postgresql.org/pgsql-bugs/2009-10/msg00024.php <nowiki>BUG #5103: "pg_ctl -w (re)start" fails with custom unix_socket_directory</nowiki>]
}}

{{TodoItem
|Modify pg_ctl behavior and exit codes to make it easier to write an LSB conforming init script
|It may be desirable to condition some of the changes on a command-line switch, to avoid breaking existing scripts. A Linux shell (sh) script is referenced which has been tested and seems to provide a high degree of conformance in multiple environments. Study of this script might suggest areas where pg_ctl could be modified to make writing an LSB conforming script easier; however, some aspects of that script would be unnecessary with other suggested changes to pg_ctl, and discussion on the lists did not reach consensus on support for all aspects of this script. Further discussion of particular changes is needed before beginning any work.
* [[Lsb_conforming_init_script|LSB conforming init script]]
These threads should be studied for other ideas on improvements:
* [http://archives.postgresql.org/pgsql-hackers/2009-08/msg01390.php <nowiki>We should Axe /contrib/start-scripts</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-08/msg01843.php <nowiki>Linux LSB init script</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg00008.php <nowiki>Re: Linux LSB init script</nowiki>]
}}

{{TodoItem
|Improve pg_ctl's detection of running postmasters
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg00000.php
* http://archives.postgresql.org/pgsql-committers/2011-06/msg00001.php
}}

{{TodoEndSubsection}}

=== psql ===
{{TodoSubsection}}

{{TodoItem
|Have psql \ds show all sequences and their settings
* [http://archives.postgresql.org/pgsql-hackers/2008-07/msg00916.php <nowiki>Re: TODO item: Have psql show current values for a sequence</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-12/msg00401.php <nowiki>Quick patch: Display sequence owner</nowiki>]
}}

{{TodoItemDone
|Have \d on a sequence indicate if the sequence is owned by a table}}

{{TodoItem
|Move psql backslash database information into the backend, use mnemonic commands?
|This would allow non-psql clients to pull the same information out of the database as psql.
* [http://archives.postgresql.org/pgsql-hackers/2004-01/msg00191.php <nowiki>Re: psql \d option list overloaded</nowiki>]
}}

{{TodoItem
|Make psql's \d commands more consistent in their handling of schemas
* [http://archives.postgresql.org/pgsql-hackers/2004-11/msg00014.php <nowiki>Re: psql and schemas</nowiki>]
}}

{{TodoItem
|Make psql's \d commands distinguish default privileges from no privileges
|ACL displays were visibly different for the two cases before we "improved" them by using array_to_string.
* [http://archives.postgresql.org/pgsql-bugs/2011-05/msg00082.php <nowiki>BUG #6021: There is no difference between default and empty access privileges with \dp</nowiki>]
}}

{{TodoItem
|Consistently display privilege information for all objects in psql}}

{{TodoItemDone
|Add "auto" expanded mode that outputs in expanded format if "wrapped" mode can't wrap the output to the screen width
|Consider using auto-expanded mode for backslash commands like \df+.
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00417.php <nowiki>Re: psql wrapped format default for backslash-d commands</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg01638.php
}}

{{TodoItem
|Prevent tab completion of SET TRANSACTION from querying the database and therefore preventing the transaction isolation level from being set.
|Currently SET <tab> causes a database lookup to check all supported session variables. This query causes problems because setting the transaction isolation level must be the first statement of a transaction.}}

{{TodoItemEasy
|\s without arguments (display history) fails with libedit, doesn't use pager either
* [http://archives.postgresql.org/pgsql-bugs/2011-06/msg00114.php <nowiki> psql \s not working - OS X</nowiki>]
}}

{{TodoItem
|Add a \set variable to control whether \s displays line numbers
|Another option is to add \# which lists line numbers, and allows command execution.
* [http://archives.postgresql.org/pgsql-hackers/2006-12/msg00255.php <nowiki>Re: psql possible TODO</nowiki>]
}}

{{TodoItem
|Include the symbolic SQLSTATE name in verbose error reports
* [http://archives.postgresql.org/pgsql-general/2007-09/msg00438.php <nowiki>Re: Checking is TSearch2 query is valid</nowiki>]
}}

{{TodoItem
|Add prompt escape to display the client and server versions
* [http://archives.postgresql.org/pgsql-hackers/2009-05/msg00310.php <nowiki>WIP patch for TODO Item: Add prompt escape to display the client and server versions</nowiki>]
}}

{{TodoItem
|Add option to wrap column values at whitespace boundaries, rather than chopping them at a fixed width.
|Currently, "wrapped" format chops values into fixed widths. Perhaps the word wrapping could use the same algorithm documented in the W3C specification.
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00404.php <nowiki>Re: psql wrapped format default for backslash-d commands</nowiki>]
* http://www.w3.org/TR/CSS21/tables.html#auto-table-layout}}

{{TodoItem
|Support the ReST table output format
|Details about the ReST format: http://docutils.sourceforge.net/rst.html#reference-documentation
* [http://archives.postgresql.org/pgsql-hackers/2008-08/msg01007.php <nowiki>Proposal: new border setting in psql</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-01/msg00518.php <nowiki>Re: Proposal: new border setting in psql</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-01/msg00609.php <nowiki>Re: Proposal: new border setting in psql</nowiki>]
}}

{{TodoItem
|Add option to print advice for people familiar with other databases
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01845.php <nowiki>MySQL-ism help patch for psql</nowiki>]
}}

{{TodoItemDone
|\dd is missing comments for several types of objects
|Comments are not handled at all for some object types, and are handled by both \dd and the individual backslash command for others. Consider a system view like pg_comments to manage this mess.
* [http://archives.postgresql.org/pgsql-general/2009-09/msg00199.php <nowiki>comment on constraint</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2010-09/msg01080.php <nowiki>pg_comments</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2011-05/msg00885.php <nowiki>patch: Allow \dd to show constraint comments</nowiki>]
}}

{{TodoItem
|Add ability to edit views with \ev
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg00023.php <nowiki>Adding \ev view editor?</nowiki>]
}}

{{TodoItem
|Fix FETCH_COUNT to handle SELECT ... INTO and WITH queries
* http://archives.postgresql.org/pgsql-hackers/2010-05/msg01565.php
* http://archives.postgresql.org/pgsql-bugs/2010-05/msg00192.php
}}

{{TodoItem
|Prevent psql from sending remaining single-line multi-statement queries after reconnecting
* http://archives.postgresql.org/pgsql-bugs/2010-05/msg00159.php
* http://archives.postgresql.org/pgsql-hackers/2010-05/msg01283.php
}}

{{TodoItemEasy
|Add \i option to bring in the specified file as a quoted literal
|This would be useful for creating functions and other areas. Details still need to be worked out.
* http://archives.postgresql.org/pgsql-bugs/2011-02/msg00016.php
* http://archives.postgresql.org/pgsql-bugs/2011-02/msg00020.php
}}

{{TodoItem
|Consider having psql -c read .psqlrc, for consistency
|psql -f already reads .psqlrc
}}

{{TodoItem
|Allow processing of multiple -f (file) options
}}

{{TodoItem
|Improve line drawing characters
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00386.php
}}

{{TodoItem
|Consider improving the continuation prompt
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg01772.php
}}

{{TodoEndSubsection}}

=== pg_dump / pg_restore ===
{{TodoSubsection}}

{{TodoItemEasy
|<nowiki>Add full object name to the tag field. eg. for operators we need '=(integer, integer)', instead of just '='.</nowiki>}}

{{TodoItem
|Add pg_dumpall custom format dumps?
* [http://archives.postgresql.org/pgsql-general/2010-05/msg00509.php pg_dumpall custom format]
|}}

{{TodoItem
|Avoid using platform-dependent locale names in pg_dumpall output
|Using native locale names puts roadblocks in the way of porting a dump to another platform. One possible solution is to get
CREATE DATABASE to accept some agreed-on set of locale names and fix them up to meet the platform's requirements.
* http://archives.postgresql.org/message-id/21396.1241716688@sss.pgh.pa.us
}}

{{TodoItem
|Allow selection of individual object(s) of all types, not just tables}}

{{TodoItem
|In a selective dump, allow dumping of an object and all its dependencies}}

{{TodoItem
|Add options like pg_restore -l and -L to pg_dump}}

{{TodoItem
|Add support for multiple pg_restore -t options, like pg_dump
|pg_restore's -t switch is less useful than pg_dump's in quite a few ways: no multiple switches, no pattern matching, no ability to pick up indexes and other dependent items for a selected table. It should be made to handle this switch just like pg_dump does.}}

{{TodoItem
|Stop dumping CASCADE on DROP TYPE commands in clean mode}}

{{TodoItem
|Allow pg_dump --clean to drop roles that own objects or have privileges
|tgl says: if this is about pg_dumpall, it's done as of 8.4. If it's really about pg_dump, what does it mean? pg_dump has no business dropping roles.}}

{{TodoItem
|Allow pg_dump to utilize multiple CPUs and I/O channels by dumping multiple objects simultaneously
|The difficulty with this is getting multiple dump processes to produce a single dump output file. It also would require several sessions to share the same snapshot.
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00205.php <nowiki>pg_dump additional options for performance</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00135.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00040.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg02454.php
}}

{{TodoItem
|Allow pg_restore to load different parts of the COPY data for a single table simultaneously}}

{{TodoItem
|Remove support for dumping from pre-7.3 servers
|In 7.3 and later, we can get accurate dependency information from the server. pg_dump still contains a lot of crufty code
to try to deal with the lack of dependency info in older servers, but the usefulness of maintaining that code grows small.}}

{{TodoItemDone
|Allow pre/data/post files when schema and data are dumped separately, for performance reasons
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00205.php <nowiki>pg_dump additional options for performance</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-07/msg00185.php <nowiki>Re: pg_dump additional options for performance</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg00821.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00135.php
}}

{{TodoItem
|Refactor handling of database attributes between pg_dump and pg_dumpall
|Currently only pg_dumpall emits database attributes, such as ALTER DATABASE SET commands and database-level GRANTs.
Many people wish that pg_dump would do that. One proposal is to let pg_dump issue such commands if the -C switch was used,
but it's unclear whether that will satisfy the demand.
* [http://archives.postgresql.org/pgsql-hackers/2008-06/msg01031.php <nowiki>ALTER DATABASE vs pg_dump</nowiki>]
* [http://archives.postgresql.org/pgsql-bugs/2010-05/msg00010.php summary of the issues]
}}

{{TodoItem
|Change pg_dump so that a comment on the dumped database is applied to the loaded database, even if the database has a different name.
|This will require new backend syntax, perhaps COMMENT ON CURRENT DATABASE. This is related to the previous item.}}

{{TodoItem
|Allow parallel restore of tar dumps
* [http://archives.postgresql.org/pgsql-hackers/2009-02/msg01154.php <nowiki>Re: parallel restore</nowiki>]
}}

{{TodoItemDone
|Allow pg_dumpall to output restorable ALTER USER/DATABASE SET settings
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00916.php
* http://archives.postgresql.org/pgsql-hackers/2011-01/msg00394.php
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg02359.php
* http://archives.postgresql.org/pgsql-hackers/2011-10/msg00489.php
}}

{{TodoEndSubsection}}

=== ecpg ===
{{TodoSubsection}}

{{TodoItem
|Docs
|Document differences between ecpg and the SQL standard and information about the Informix-compatibility module.}}

{{TodoItem
|Solve cardinality > 1 for input descriptors / variables?}}

{{TodoItem
|Add a semantic check level, e.g. check if a table really exists}}

{{TodoItem
|fix handling of DB attributes that are arrays}}

{{TodoItem
|Fix nested C comments}}

{{TodoItemEasy
|sqlwarn[6] should be 'W' if the PRECISION or SCALE value specified}}

{{TodoItem
|Make SET CONNECTION thread-aware, non-standard?}}

{{TodoItem
|Allow multidimensional arrays}}

{{TodoItem
|Implement COPY FROM STDIN}}

{{TodoItem
|Provide a way to specify size of a bytea parameter
* [http://archives.postgresql.org/message-id/200906192131.n5JLVoMo044178@wwwmaster.postgresql.org <nowiki>BUG #4866: ECPG and BYTEA</nowiki>]
}}

{{TodoItemEasy
|Fix small memory leaks in ecpg
|Memory leaks in a short running application like ecpg are not really a problem, but make debugging more complicated}}

{{TodoItem
|Allow reuse of cursor name variables
* [http://archives.postgresql.org/message-id/20100329113435.GA3430@feivel.credativ.lan <nowiki>Problems with variable cursorname in ecpg</nowiki>]
}}

{{TodoEndSubsection}}

=== libpq ===
{{TodoSubsection}}

{{TodoItem
|Prevent PQfnumber() from lowercasing unquoted column names
|PQfnumber() should never have been doing lowercasing, but historically it has so we need a way to prevent it}}

{{TodoItem
|Allow statement results to be automatically batched to the client
|Currently all statement results are transferred to the libpq client before libpq makes the results available to the application. This feature would allow the application to make use of the first result rows while the rest are transferred, or held on the server waiting for them to be requested by libpq. One complexity is that a statement like SELECT 1/col could error out mid-way through the result set.}}

{{TodoItem
|Consider disallowing multiple queries in PQexec() as an additional barrier to SQL injection attacks
* [http://archives.postgresql.org/pgsql-hackers/2007-01/msg00184.php <nowiki>Re: InitPostgres and flatfiles question</nowiki>]
}}

{{TodoItem
|Add PQexecf() that allows complex parameter substitution
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg01803.php <nowiki>Last minute mini-proposal (I know, know) for PQexecf()</nowiki>]
}}

{{TodoItem
|Add SQLSTATE and severity to errors generated within libpq itself
* [http://archives.postgresql.org/pgsql-interfaces/2007-11/msg00015.php <nowiki>v8.1: Error severity on libpq PGconn*</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg01425.php
}}

{{TodoItem
|Add support for interface/ipaddress binding to libpq
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01811.php <nowiki>SR/libpq - outbound interface/ipaddress binding</nowiki>]
}}

{{TodoEndSubsection}}

== Triggers ==

{{TodoItem
|Improve storage of deferred trigger queue
|Right now all deferred trigger information is stored in backend memory. This could exhaust memory for very large trigger queues. This item involves dumping large queues into files, or doing some kind of join to process all the triggers, some bulk operation, or a bitmap.
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00876.php <nowiki>Re: BUG #4204: COPY to table with FK has memory leak</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-10/msg00464.php <nowiki>Scaling up deferred unique checks and the after trigger queue</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2011-08/msg00023.php
}}

{{TodoItem
|Allow triggers to be disabled in only the current session.
|This is currently possible by starting a multi-statement transaction, modifying the system tables, performing the desired SQL, restoring the system tables, and committing the transaction. ALTER TABLE ... TRIGGER requires a table lock so it is not ideal for this usage.}}

{{TodoItem
|With disabled triggers, allow pg_dump to use ALTER TABLE ADD FOREIGN KEY
|If the dump is known to be valid, allow foreign keys to be added without revalidating the data.}}

{{TodoItem
|Allow statement-level triggers to access modified rows}}

{{TodoItem
|When statement-level triggers are defined on a parent table, have them fire only on the parent table, and fire child table triggers only where appropriate
* [http://archives.postgresql.org/pgsql-hackers/2008-11/msg01883.php <nowiki>Statement-level triggers and inheritance</nowiki>]
}}

{{TodoItem
|Allow AFTER triggers on system tables
|System tables are modified in many places in the backend without going through the executor and therefore not causing triggers to fire. To complete this item, the functions that modify system tables will have to fire triggers.
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01665.php
* http://wiki.postgresql.org/wiki/DDL_Triggers
* http://archives.postgresql.org/pgsql-hackers/2011-12/msg00022.php
}}

{{TodoItem
|Tighten trigger permission checks
* [http://archives.postgresql.org/pgsql-hackers/2006-12/msg00564.php <nowiki>Security leak with trigger functions?</nowiki>]
}}

{{TodoItem
|Allow BEFORE INSERT triggers on views
* [http://archives.postgresql.org/pgsql-general/2007-02/msg01466.php <nowiki>Re: Why can't I put a BEFORE EACH ROW trigger on a view?</nowiki>]
}}

{{TodoItem
|Add database and transaction-level triggers
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00451.php <nowiki>Proposal for db level triggers</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00620.php <nowiki>triggers on prepare, commit, rollback... ?</nowiki>]
}}

{{TodoItem
|Reduce locking requirements for creating a trigger
* [http://archives.postgresql.org/pgsql-hackers/2008-06/msg00635.php <nowiki>Re: Change lock requirements for adding a trigger</nowiki>]
}}

{{TodoItem
|Avoid requirement for "AFTER" trigger functions to return a value
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg02384.php
}}

{{TodoItem
|Allow creation of inline triggers
* http://archives.postgresql.org/pgsql-hackers/2012-02/msg00708.php
}}

== Inheritance ==

{{TodoItem
|Allow inherited tables to inherit indexes, UNIQUE constraints, and primary/foreign keys
* [http://archives.postgresql.org/pgsql-hackers/2010-05/msg00285.php <nowiki>Partitioning/inherited tables vs FKs</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00039.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00305.php
}}

{{TodoItem
|Honor UNIQUE INDEX on base column in INSERTs/UPDATEs on inherited table, e.g. INSERT INTO inherit_table (unique_index_col) VALUES (dup) should fail
|The main difficulty with this item is the problem of creating an index that can span multiple tables.}}

{{TodoItem
|Determine whether ALTER TABLE / SET SCHEMA should work on inheritance hierarchies (and thus support ONLY). If yes, implement it.}}

{{TodoItem
|ALTER TABLE variants sometimes support recursion and sometimes not, but this is poorly/not documented, and the ONLY marker would then be silently ignored. Clarify the documentation, and reject ONLY if it is not supported.}}

== Indexes ==

{{TodoItem
|Prevent index uniqueness checks when UPDATE does not modify the column
|Uniqueness (index) checks are done when updating a column even if the column is not modified by the UPDATE.
However, HOT already short-circuits this in common cases, so more work might not be helpful.}}

{{TodoItem
|Allow the creation of on-disk bitmap indexes which can be quickly combined with other bitmap indexes
|Such indexes could be more compact if there are only a few distinct values. Such indexes can also be compressed. Keeping such indexes updated can be costly.
* [http://archives.postgresql.org/pgsql-patches/2005-07/msg00512.php <nowiki>Re: Bitmap index AM</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2006-12/msg01107.php <nowiki>Bitmap index thoughts</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg00265.php <nowiki>Stream bitmaps</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg01214.php <nowiki>Re: Bitmapscan changes - Requesting further feedback</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2007-05/msg00013.php <nowiki>Updated bitmap index patch</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-07/msg00741.php <nowiki>Reviewing new index types (was Re: [PATCHES] Updated bitmap indexpatch)</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg01023.php <nowiki>Bitmap Indexes: request for feedback</nowiki>]
* http://archives.postgresql.org/message-id/800923.27831.qm@web29010.mail.ird.yahoo.com
}}

{{TodoItem
|Allow accurate statistics to be collected on indexes with more than one column or expression indexes, perhaps using per-index statistics
* [http://archives.postgresql.org/pgsql-performance/2006-10/msg00222.php <nowiki>Re: Simple join optimized badly?</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg01131.php <nowiki>Stats for multi-column indexes</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg00741.php <nowiki>Cross-column statistics revisited</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-06/msg01431.php <nowiki>Multi-Dimensional Histograms</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00913.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg02179.php
* http://archives.postgresql.org/pgsql-hackers/2011-01/msg00459.php
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg02054.php
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg01731.php
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg00894.php
* http://archives.postgresql.org/pgsql-hackers/2011-09/msg00679.php
}}

{{TodoItem
|Consider having a larger statistics target for indexed columns and expression indexes.
}}

{{TodoItem
|Consider smaller indexes that record a range of values per heap page, rather than having one index entry for every heap row
|This is useful if the heap is clustered by the indexed values.
* [http://archives.postgresql.org/pgsql-hackers/2006-12/msg00341.php <nowiki>Grouped Index Tuples</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-02/msg01264.php <nowiki>Grouped Index Tuples</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg00465.php <nowiki>Grouped Index Tuples / Clustered Indexes</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2007-03/msg00163.php <nowiki>Bitmapscan changes</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-08/msg00014.php <nowiki>Re: GIT patch</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-08/msg00487.php <nowiki>Re: Index Tuple Compression Approach?</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-04/msg01589.php <nowiki>Re: Index AM change proposals, redux</nowiki>]
}}

{{TodoItem
|Add REINDEX CONCURRENTLY, like CREATE INDEX CONCURRENTLY
|This is difficult because you must upgrade to an exclusive table lock to replace the existing index file. CREATE INDEX CONCURRENTLY does not have this complication. This would allow index compaction without downtime.
* [http://archives.postgresql.org/pgsql-performance/2007-08/msg00289.php <nowiki>Re: When/if to Reindex</nowiki>]
}}

{{TodoItem
|Allow multiple indexes to be created concurrently, ideally via a single heap scan
|pg_restore allows parallel index builds, but it is done via subprocesses, and there is no SQL interface for this.
* http://archives.postgresql.org/pgsql-performance/2011-04/msg00093.php
}}

{{TodoItem
|Consider sorting entries before inserting into btree index
* [http://archives.postgresql.org/pgsql-general/2008-01/msg01010.php <nowiki>Re: ATTN: Clodaldo was Performance problem. Could it be related to 8.3-beta4?</nowiki>]
}}

{{TodoItem
|Allow index scans to return matching index keys, not just the matching heap locations
* [http://archives.postgresql.org/pgsql-hackers/2008-04/msg01657.php <nowiki>Re: Is this TODO item done?</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-08/msg01477.php <nowiki>Index-only quals</nowiki>]
}}

{{TodoItem
|Allow creation of an index that can do comparisons to test if a value is between two column values
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00757.php <nowiki>Proposal: temporal extension "period" data type</nowiki>]
}}

{{TodoItem
|Consider using "effective_io_concurrency" for index scans
* Currently only bitmap scans use this, which might be fine because most multi-row index scans use bitmap scans.
}}

{{TodoItem
|Fix problem with btree page splits during checkpoints
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg00052.php
* http://archives.postgresql.org/pgsql-hackers/2011-09/msg00184.php
}}

=== GIST ===
{{TodoSubsection}}

{{TodoItem
|Add more GIST index support for geometric data types}}

{{TodoItem
|Allow GIST indexes to create certain complex index types, like digital trees (see Aoki)}}

{{TodoItem
|Fix performance issues in contrib/seg and contrib/cube GiST support
* [http://archives.postgresql.org/message-id/alpine.DEB.2.00.0904161633160.4053@aragorn.flymine.org GiST index performance]
* [http://archives.postgresql.org/message-id/alpine.DEB.2.00.0904221704470.22330@aragorn.flymine.org draft patch]
* [http://archives.postgresql.org/pgsql-performance/2009-05/msg00069.php <nowiki>Re: GiST index performance</nowiki>]
* [http://archives.postgresql.org/pgsql-performance/2009-06/msg00068.php <nowiki>GiST index performance</nowiki>]
}}

{{TodoEndSubsection}}

=== Hash ===
{{TodoSubsection}}

{{TodoItem
|Add UNIQUE capability to hash indexes}}

{{TodoItem
|Add hash WAL logging for crash recovery
* http://archives.postgresql.org/pgsql-performance/2011-09/msg00196.php
}}

{{TodoItem
|Allow multi-column hash indexes}}

{{TodoEndSubsection}}

== Sorting ==

{{TodoItem
|Consider whether duplicate keys should be sorted by block/offset
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00558.php <nowiki>Remove hacks for old bad qsort() implementations?</nowiki>]
}}

{{TodoItem
|Consider being smarter about memory and external files used during sorts
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg01101.php <nowiki>Sorting Improvements for 8.4</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-12/msg00045.php <nowiki>Re: Sorting Improvements for 8.4</nowiki>]
}}

{{TodoItem
|Consider detoasting keys before sorting}}

{{TodoItem
|Allow sorts to use more available memory
* http://archives.postgresql.org/pgsql-hackers/2007-11/msg01026.php
* http://archives.postgresql.org/pgsql-hackers/2010-09/msg01123.php
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg01957.php
}}

== Fsync ==

{{TodoItem
|Determine optimal fdatasync/fsync, O_SYNC/O_DSYNC options and whether fsync does anything
|Ideally this requires a separate test program like /contrib/pg_test_fsync that can be run at initdb time or optionally later.}}

{{TodoItem
|Consider sorting writes during checkpoint
* [http://archives.postgresql.org/pgsql-hackers/2007-06/msg00541.php <nowiki>Sorted writes in checkpoint</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-07/msg00050.php <nowiki>Re: Sorting writes during checkpoint</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-10/msg02012.php
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg00278.php
* [http://archives.postgresql.org/message-id/CA+TgmoaHu1zuNohoE=cEP0nSc+0wtuRSyEAj_Af2XhxU+ry6-w@mail.gmail.com checkpoint writeback via sync_file_range]
}}

== Cache Usage ==

{{TodoItemDone
|Speed up COUNT(*)
|We could use a fixed row count and a +/- count to follow MVCC visibility rules, or a single cached value could be used and invalidated if anyone modifies the table. Another idea is to get a count directly from a unique index, but for this to be faster than a sequential scan it must avoid access to the heap to obtain tuple visibility information. Note that the index-only scans feature is now implemented which now dramatically speeds up some COUNT(*) cases.
* http://wiki.postgresql.org/wiki/Slow_Counting
}}

{{TodoItem
|Provide a way to calculate an "estimated COUNT(*)"
|Perhaps by using the optimizer's cardinality estimates or random sampling.
* [http://archives.postgresql.org/pgsql-hackers/2005-11/msg00943.php <nowiki>Re: Improving count(*)</nowiki>]
* http://wiki.postgresql.org/wiki/Slow_Counting
}}

{{TodoItemDone
|Allow data to be pulled directly from indexes
|Currently indexes do not have enough tuple visibility information to allow data to be pulled from the index without also accessing the heap. The idea is to use the visibility map used for vacuum to avoid heap lookups on pages where all tuples are visible.
* [http://wiki.postgresql.org/wiki/Index-only_scans Index-Only Scans wiki]
}}

{{TodoItem
|Consider automatic caching of statements at various levels:
* Parsed query tree
* Query execute plan
* Query results

:
* [http://archives.postgresql.org/pgsql-hackers/2008-04/msg00823.php <nowiki>Cached Query Plans (was: global prepared statements)</nowiki>]
}}

{{TodoItem
|Consider increasing internal areas (NUM_CLOG_BUFFERS) when shared buffers is increased
* [http://archives.postgresql.org/pgsql-hackers/2005-10/msg01419.php <nowiki>Re: slru.c race condition (was Re: TRAP: FailedAssertion("!((itemid)->lp_flags & 0x01)",)</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-08/msg00030.php <nowiki>clog_buffers to 64 in 8.3?</nowiki>]
* [http://archives.postgresql.org/pgsql-performance/2007-08/msg00024.php <nowiki>CLOG Patch</nowiki>]
}}

{{TodoItem
|Consider decreasing the amount of memory used by PrivateRefCount
|
* [http://archives.postgresql.org/pgsql-hackers/2006-11/msg00797.php <nowiki>PrivateRefCount (for 8.3)</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-01/msg00752.php <nowiki>Re: PrivateRefCount (for 8.3)</nowiki>]
}}

{{TodoItem
|Consider allowing higher priority queries to have referenced buffer cache pages stay in memory longer
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg00562.php <nowiki>Re: How to keep a table in memory?</nowiki>]
}}

== Vacuum ==

{{TodoItem
|Auto-fill the free space map by scanning the buffer cache or by checking pages written by the background writer
* [http://archives.postgresql.org/pgsql-hackers/2006-02/msg01125.php <nowiki>Dead Space Map</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2006-03/msg00011.php <nowiki>Re: Automatic free space map filling</nowiki>]
}}

{{TodoItem
|Allow concurrent inserts to use recently created pages rather than creating new ones
* http://archives.postgresql.org/pgsql-hackers/2010-05/msg00853.php
}}

{{TodoItem
|Consider having single-page pruning update the visibility map
* <nowiki>https://commitfest.postgresql.org/action/patch_view?id=75</nowiki>
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg02344.php <nowiki>Re: visibility maps and heap_prune</nowiki>]
}}

{{TodoItem
|Improve tracking of total relation tuple counts now that vacuum doesn't always scan the whole heap
* [http://archives.postgresql.org/pgsql-hackers/2009-06/msg00531.php Partial vacuum versus pg_class.reltuples]
}}

{{TodoItem
|Bias FSM towards returning free space near the beginning of the heap file, in hopes that empty pages at the end can be truncated by VACUUM
* [http://archives.postgresql.org/pgsql-hackers/2009-09/msg01124.php <nowiki>FSM search modes</nowiki>]
}}

{{TodoItem
|Consider a more compact data representation for dead tuple locations within VACUUM
* [http://archives.postgresql.org/pgsql-patches/2007-05/msg00143.php <nowiki>Re: Have vacuum emit a warning when it runs out of maintenance_work_mem</nowiki>]
}}

{{TodoItem
|Provide more information in order to improve user-side estimates of dead space bloat in relations
* [http://archives.postgresql.org/pgsql-general/2009-05/msg01039.php <nowiki>Re: Bloated Table</nowiki>]
}}

{{TodoItem
|Improve locking behaviour of vacuum during trailing page truncation
* http://archives.postgresql.org/pgsql-bugs/2011-03/msg00319.php
* http://archives.postgresql.org/message-id/4D8DF88E.7080205@Yahoo.com
}}

{{TodoItem
|Reduce the number of table scans performed by vacuum
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg01119.php
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg00605.php
* http://archives.postgresql.org/pgsql-hackers/2011-07/msg00624.php
}}

=== Auto-vacuum ===
{{TodoSubsection}}

{{TodoItemEasy
|Issue log message to suggest VACUUM FULL if a table is nearly empty?}}

{{TodoItem
|Prevent long-lived temporary tables from causing frozen-xid advancement starvation
|The problem is that autovacuum cannot vacuum them to set frozen xids; only the session that created them can do that.
* [http://archives.postgresql.org/pgsql-general/2007-06/msg01645.php <nowiki>Re: AutoVacuum Behaviour Question</nowiki>]
}}

{{TodoItem
|Prevent autovacuum from running if an old transaction is still running from the last vacuum
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg00899.php <nowiki>Re: Autovacuum and OldestXmin</nowiki>]
}}

{{TodoItem
|Have autoanalyze of parent tables occur when child tables are modified
* http://archives.postgresql.org/pgsql-performance/2010-06/msg00137.php
* http://archives.postgresql.org/pgsql-performance/2010-10/msg00271.php
}}

{{TodoItem
|Allow parallel cores to be used by vacuumdb
* [http://archives.postgresql.org/message-id/4F10A728.7090403@agliodbs.com vacuumdb -j]
}}

{{TodoEndSubsection}}

== Locking ==

{{TodoItem
|Fix priority ordering of read and write light-weight locks
* [http://archives.postgresql.org/pgsql-hackers/2004-11/msg00893.php <nowiki>lwlocks and starvation</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2004-11/msg00905.php <nowiki>Re: lwlocks and starvation</nowiki>]
}}

{{TodoItem
|Fix problem when multiple subtransactions of the same outer transaction hold different types of locks, and one subtransaction aborts
* [http://archives.postgresql.org/pgsql-hackers/2006-11/msg01011.php <nowiki>FOR SHARE vs FOR UPDATE locks</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2006-12/msg00001.php <nowiki>Re: FOR SHARE vs FOR UPDATE locks</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-02/msg00435.php <nowiki>Re: [PATCHES] [pgsql-patches] Phantom Command IDs, updated patch</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-05/msg00773.php <nowiki>Re: savepoints and upgrading locks</nowiki>]
}}

{{TodoItem
|Allow UPDATEs on only non-referential integrity columns not to conflict with referential integrity locks
* [http://archives.postgresql.org/pgsql-hackers/2007-02/msg00073.php <nowiki>Referential Integrity and SHARE locks</nowiki>]
}}

{{TodoItem
|Add idle_in_transaction_timeout GUC so locks are not held for long periods of time}}

{{TodoItem
|Improve deadlock detection when a page cleaning lock conflicts with a shared buffer that is pinned
* [http://archives.postgresql.org/pgsql-bugs/2008-01/msg00138.php <nowiki>BUG #3883: Autovacuum deadlock with truncate?</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00873.php <nowiki>Thoughts about bug #3883</nowiki>]
* [http://archives.postgresql.org/pgsql-committers/2008-01/msg00365.php <nowiki>Re: pgsql: Add checks to TRUNCATE, CLUSTER, and REINDEX to prevent</nowiki>]
}}

{{TodoItem
|Detect deadlocks involving LockBufferForCleanup()
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00873.php <nowiki>Thoughts about bug #3883</nowiki>]
}}

{{TodoItem
|Allow finer control over who is cancelled in a deadlock
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01727.php
}}

{{TodoItem
|Consider a lock timeout parameter
* [http://archives.postgresql.org/pgsql-hackers/2009-05/msg00485.php <nowiki>SELECT ... FOR UPDATE [WAIT integer | NOWAIT] for 8.5</nowiki>]
}}

{{TodoItemDone
|Reduce number of unnecessary false positives in Serializable Snapshot Isolation
* [http://archives.postgresql.org/pgsql-hackers/2011-06/msg00609.php <nowiki>SSI heap_insert and page-level predicate locks</nowiki>]
}}

== Startup Time Improvements ==

{{TodoItem
|Experiment with multi-threaded backend for backend creation
|This would prevent the overhead associated with process creation. Most operating systems have trivial process creation time compared to database startup overhead, but a few operating systems (Win32, Solaris) might benefit from threading. Also explore the idea of a single session using multiple threads to execute a statement faster.}}

{{TodoItem
|Allow backends to change their database without restart
|This allows for faster server startup.
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg00843.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00336.php
}}

== Write-Ahead Log ==

{{TodoItem
|Eliminate need to write full pages to WAL before page modification
|Currently, to protect against partial disk page writes, we write full page images to WAL before they are modified so we can correct any partial page writes during recovery. These pages can also be eliminated from point-in-time archive files.
* [http://archives.postgresql.org/pgsql-hackers/2002-06/msg00655.php <nowiki>Re: Index Scans become Seq Scans after VACUUM ANALYSE</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg01191.php
* [http://archives.postgresql.org/message-id/20120105061916.GB21048@fetter.org WIP double writes]
* [http://archives.postgresql.org/message-id/4EFC449F02000025000441CD@gw.wicourts.gov double writes]
* [http://archives.postgresql.org/message-id/20120110214344.GB21106@fetter.org Double-write with Fast Checksums]
* [http://archives.postgresql.org/message-id/1962493974.656458.1327703514780.JavaMail.root@zimbra-prod-mbox-4.vmware.com double writes using "double-write buffer" approach]
}}

{{TodoItem
|When full page writes are off, write CRC to WAL and check file system blocks on recovery
|If CRC check fails during recovery, remember the page in case a later CRC for that page properly matches. The difficulty is that hint bits are not WAL logged, meaning a valid page might not match the earlier CRC.}}

{{TodoItem
|Write full pages during file system write and not when the page is modified in the buffer cache
|This allows most full page writes to happen in the background writer. It might cause problems for applying WAL on recovery into a partially-written page, but later the full page will be replaced from WAL.
* [http://archives.postgresql.org/message-id/CAGvK12UST-tPhyLrSLuSpwFxZbAO79yYrhV2xaLmS2MkUxNUVQ@mail.gmail.com Page Checksums + Double Writes]
}}

{{TodoItem
|Reduce WAL traffic so only modified values are written rather than entire rows
* [http://archives.postgresql.org/pgsql-hackers/2007-03/msg01589.php <nowiki>Reduction in WAL for UPDATEs</nowiki>]
}}

{{TodoItem
|Allow WAL information to recover corrupted pg_controldata
* [http://archives.postgresql.org/pgsql-patches/2006-06/msg00025.php <nowiki>Re: [HACKERS] pg_resetxlog -r flag</nowiki>]
}}

{{TodoItem
|Find a way to reduce rotational delay when repeatedly writing last WAL page
|Currently fsync of WAL requires the disk platter to perform a full rotation to fsync again. One idea is to write the WAL to different offsets that might reduce the rotational delay.
* [http://archives.postgresql.org/pgsql-hackers/2002-11/msg00483.php <nowiki>500 tpsQL + WAL log implementation</nowiki>]
}}

{{TodoItem
|Speed WAL recovery by allowing more than one page to be prefetched
|This should be done utilizing the same infrastructure used for prefetching in general to avoid introducing complex error-prone code in WAL replay.
* [http://archives.postgresql.org/pgsql-general/2007-12/msg00683.php <nowiki>Slow PITR restore</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-12/msg00497.php <nowiki>Re: [GENERAL] Slow PITR restore</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg01279.php <nowiki>Read-ahead and parallelism in redo recovery</nowiki>]
}}

{{TodoItem
|Improve WAL concurrency by increasing lock granularity
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00556.php <nowiki>Reworking WAL locking</nowiki>]
}}

{{TodoItem
|Be more aggressive about creating WAL files
* [http://archives.postgresql.org/pgsql-hackers/2007-10/msg01325.php <nowiki>Re: PANIC caused by open_sync on Linux</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2004-07/msg01075.php <nowiki>PreallocXlogFiles</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2005-04/msg00556.php <nowiki>WAL/PITR additional items</nowiki>]
}}

{{TodoItem
|Have resource managers report the duration of their status changes
* [http://archives.postgresql.org/pgsql-hackers/2007-10/msg01468.php <nowiki>Recovery of Multi-stage WAL actions</nowiki>]
}}

{{TodoItem
|Move pgfoundry's xlogdump to /contrib and have it rely more closely on the WAL backend code
* [http://archives.postgresql.org/pgsql-hackers/2007-11/msg00035.php <nowiki>xlogdump</nowiki>]
}}

{{TodoItem
|Close deleted WAL files held open in *nix by long-lived read-only backends
* [http://archives.postgresql.org/pgsql-hackers/2009-11/msg01754.php <nowiki>Deleted WAL files held open by backends in Linux</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-12/msg00060.php <nowiki>Re: Deleted WAL files held open by backends in Linux</nowiki>]
}}

== Optimizer / Executor ==

{{TodoItem
|Improve selectivity functions for geometric operators}}

{{TodoItem
|Consider increasing the default values of from_collapse_limit, join_collapse_limit, and/or geqo_threshold
* [http://archives.postgresql.org/message-id/4136ffa0905210551u22eeb31bn5655dbe7c9a3aed5@mail.gmail.com from_collapse_limit vs. geqo_threshold]
}}

{{TodoItem
|Improve ability to display optimizer analysis using OPTIMIZER_DEBUG}}

{{TodoItem
|Log statements where the optimizer row estimates were dramatically different from the number of rows actually found?}}

{{TodoItem
|Consider compressed annealing to search for query plans
|This might replace GEQO.
* http://archives.postgresql.org/message-id/15658.1241278636%40sss.pgh.pa.us
}}

{{TodoItem
|Improve use of expression indexes for ORDER BY
* [http://archives.postgresql.org/pgsql-hackers/2009-08/msg01553.php <nowiki>Resjunk sort columns, Heikki's index-only quals patch, and bug #5000</nowiki>]
}}

{{TodoItem
|Modify the planner to better estimate caching effects
* http://archives.postgresql.org/pgsql-performance/2010-11/msg00117.php
}}

{{TodoItem
|Allow shared buffer cache contents to affect index cost computations
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg01140.php
}}

=== Hashing ===
{{TodoSubsection}}

{{TodoItem
|Consider using a hash for joining to a large IN (VALUES ...) list
* [http://archives.postgresql.org/pgsql-hackers/2007-05/msg00450.php <nowiki>Planning large IN lists</nowiki>]
}}

{{TodoItem
|Allow single batch hash joins to preserve outer pathkeys
* [http://archives.postgresql.org/pgsql-hackers/2008-09/msg00806.php Re: Potential Join Performance Issue]
* [http://archives.postgresql.org/pgsql-hackers/2009-04/msg00153.php a few crazy ideas about hash joins]
}}

{{TodoItem
|"lazy" hash tables - look up only the tuples that are actually requested
* [http://archives.postgresql.org/pgsql-hackers/2009-04/msg00153.php a few crazy ideas about hash joins]
}}

{{TodoItem
|Avoid building the same hash table more than once during the same query
* [http://archives.postgresql.org/pgsql-hackers/2009-04/msg00153.php a few crazy ideas about hash joins]
}}

{{TodoItem
|Avoid hashing for distinct and then re-hashing for hash join
* [http://archives.postgresql.org/message-id/4136ffa0902191346g62081081v8607f0b92c206f0a@mail.gmail.com Re: Fixing Grittner's planner issues]
* [http://archives.postgresql.org/pgsql-hackers/2009-04/msg00153.php a few crazy ideas about hash joins]
}}

{{TodoEndSubsection}}

== Background Writer ==

{{TodoItem
|Consider having the background writer update the transaction status hint bits before writing out the page
|Implementing this requires the background writer to have access to system catalogs and the transaction status log.}}

{{TodoItem
|Consider adding buffers the background writer finds reusable to the free list
* [http://archives.postgresql.org/pgsql-hackers/2007-04/msg00781.php <nowiki>Background LRU Writer/free list</nowiki>]
* [http://archives.postgresql.org/message-id/CA+U5nMKtvyDcV4zTr7bq7t6cA2nBfLxCJ8tQgVBnc5ddRPO+Bg@mail.gmail.com our buffer replacement strategy is kind of lame]
}}

{{TodoItem
|Automatically tune bgwriter_delay based on activity rather then using a fixed interval
* [http://archives.postgresql.org/pgsql-hackers/2007-04/msg00781.php <nowiki>Background LRU Writer/free list</nowiki>]
* [http://archives.postgresql.org/message-id/CA+U5nMKtvyDcV4zTr7bq7t6cA2nBfLxCJ8tQgVBnc5ddRPO+Bg@mail.gmail.com our buffer replacement strategy is kind of lame]
}}

{{TodoItem
|Consider whether increasing BM_MAX_USAGE_COUNT improves performance
* [http://archives.postgresql.org/pgsql-hackers/2007-06/msg01007.php <nowiki>Bgwriter LRU cleaning: we've been going at this all wrong</nowiki>]
}}

{{TodoItem
|Test to see if calling PreallocXlogFiles() from the background writer will help with WAL segment creation latency
* [http://archives.postgresql.org/pgsql-patches/2007-06/msg00340.php <nowiki>Re: Load Distributed Checkpoints, final patch</nowiki>]
}}

== Concurrent Use of Resources ==

{{TodoItem
|Do async I/O for faster random read-ahead of data
|Async I/O allows multiple I/O requests to be sent to the disk with results coming back asynchronously.
* [http://archives.postgresql.org/pgsql-hackers/2006-10/msg00820.php <nowiki>Asynchronous I/O Support</nowiki>]
* [http://archives.postgresql.org/pgsql-performance/2007-09/msg00255.php <nowiki>Re: random_page_costs - are defaults of 4.0 realistic for SCSI RAID 1</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-12/msg00027.php <nowiki>There's random access and then there's random access</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-01/msg00170.php <nowiki>Bitmap index scan preread using posix_fadvise (Was: There's random access and then there's random access)</nowiki>]
The above patch is already applied as of 8.4, but it still remains to figure out how to handle plain indexscans effectively.
* [http://archives.postgresql.org//pgsql-hackers/2009-01/msg00806.php Problems with the patch submitted for posix_fadvise in index scans]
}}

{{TodoItem
|Experiment with multi-threaded backend for better I/O utilization
|This would allow a single query to make use of multiple I/O channels simultaneously. One idea is to create a background reader that can pre-fetch sequential and index scan pages needed by other backends. This could be expanded to allow concurrent reads from multiple devices in a partitioned table.
* http://archives.postgresql.org/pgsql-performance/2011-02/msg00123.php
}}

{{TodoItem
|Experiment with multi-threaded backend for better CPU utilization
|This would allow several CPUs to be used for a single query, such as for sorting or query execution.
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg00945.php <nowiki>Multi CPU Queries - Feedback and/or suggestions wanted!</nowiki>]
}}

{{TodoItem
|SMP scalability improvements
* [http://archives.postgresql.org/pgsql-hackers/2007-07/msg00439.php <nowiki>Straightforward changes for increased SMP scalability</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-09/msg00206.php <nowiki>Re: Reducing Transaction Start/End Contention</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00361.php <nowiki>Re: Reducing Transaction Start/End Contention</nowiki>]
}}

== TOAST ==

{{TodoItem
|Allow user configuration of TOAST thresholds
* [http://archives.postgresql.org/pgsql-hackers/2007-02/msg00213.php <nowiki>Re: Proposed adjustments in MaxTupleSize and toastthresholds</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2007-08/msg00082.php <nowiki>pg_lzcompress strategy parameters</nowiki>]
}}

{{TodoItem
|Reduce unnecessary cases of deTOASTing
* [http://archives.postgresql.org/pgsql-hackers/2007-09/msg00895.php <nowiki>Re: [PATCHES] Eliminate more detoast copies for packed varlenas</nowiki>]
}}

{{TodoItem
|Reduce costs of repeat de-TOASTing of values
* [http://archives.postgresql.org/pgsql-hackers/2008-06/msg01096.php <nowiki>WIP patch: reducing overhead for repeat de-TOASTing</nowiki>]
}}

== Monitoring ==
{{TodoItem
|Expand pg_stat_activity for easier integration with monitoring tools
|* http://archives.postgresql.org/message-id/4DFA13A5.2060200@2ndQuadrant.com
}}

{{TodoItem
|Add column to pg_stat_activity that shows the progress of long-running commands like CREATE INDEX and VACUUM
* [http://archives.postgresql.org/pgsql-patches/2008-04/msg00203.php <nowiki>EXPLAIN progress info</nowiki>]
* The CLUSTER/VACUUM FULL implementation would also be useful to track this way
}}

{{TodoItem
|Have pg_stat_activity display query strings in the correct client encoding
* [http://archives.postgresql.org/pgsql-hackers/2009-01/msg00131.php <nowiki>pg_stats queries versus per-database encodings</nowiki>]
}}

{{TodoItemEasy
|Expose pg_controldata via an SQL interface
|Helpful for monitoring replicated databases
* http://archives.postgresql.org/message-id/4B901D73.8030003@agliodbs.com
* [http://archives.postgresql.org/message-id/4B959D7A.6010907@joeconway.com initial patch]
}}

== Miscellaneous Performance ==

{{TodoItem
|Use mmap() rather than SYSV for shared buffers?
|This would remove the requirement for SYSV SHM but would introduce portability issues. Anonymous mmap (or mmap to /dev/zero) is required to prevent I/O overhead. We could also consider mmap() for writing WAL.
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg00750.php
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg00756.php
}}

{{TodoItem
|Rather than consider mmap()-ing in 8k pages, consider mmap()'ing entire files into a backend?
|Doing I/O to large tables would consume a lot of address space or require frequent mapping/unmapping. Extending the file also causes mapping problems that might require mapping only individual pages, leading to thousands of mappings. Another problem is that there is no way to _prevent_ I/O to disk from the dirty shared buffers so changes could hit disk before WAL is written.
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01239.php
}}

{{TodoItem
|Consider ways of storing rows more compactly on disk:
* Reduce the row header size?
* Consider reducing on-disk varlena length from four bytes to two because a heap row cannot be more than 64k in length}}

{{TodoItem
|Consider transaction start/end performance improvements
* [http://archives.postgresql.org/pgsql-hackers/2007-07/msg00948.php <nowiki>Reducing Transaction Start/End Contention</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00361.php <nowiki>Re: Reducing Transaction Start/End Contention</nowiki>]
}}

{{TodoItem
|Allow configuration of backend priorities via the operating system
|Though backend priorities make priority inversion during lock waits possible, research shows that this is not a huge problem.
* [http://archives.postgresql.org/pgsql-general/2007-02/msg00493.php <nowiki>Priorities for users or queries?</nowiki>]
}}

{{TodoItem
|Consider increasing the minimum allowed number of shared buffers
* [http://archives.postgresql.org/pgsql-bugs/2008-02/msg00157.php <nowiki>Re: [PATCH] Don't bail with legitimate -N/-B options</nowiki>]
}}

{{TodoItem
|Consider if CommandCounterIncrement() can avoid its AcceptInvalidationMessages() call
* [http://archives.postgresql.org/pgsql-committers/2007-11/msg00585.php <nowiki>pgsql: Avoid incrementing the CommandCounter when</nowiki>]
}}

{{TodoItem
|Consider Cartesian joins when both relations are needed to form an indexscan qualification for a third relation
* [http://archives.postgresql.org/pgsql-performance/2007-12/msg00090.php <nowiki>Re: TB-sized databases</nowiki>]
}}

{{TodoItem
|Consider not storing a NULL bitmap on disk if all the NULLs are trailing
* [http://archives.postgresql.org/pgsql-hackers/2007-12/msg00624.php <nowiki>Proposal for Null Bitmap Optimization(for Trailing NULLs)</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2007-12/msg00109.php <nowiki>Re: [HACKERS] Proposal for Null Bitmap Optimization(for TrailingNULLs)</nowiki>]
}}

{{TodoItem
|Sort large UPDATE/DELETEs so it is done in heap order
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg01119.php <nowiki>Possible future performance improvement: sort updates/deletes by ctid</nowiki>]
}}

{{TodoItemDone
|Allow one transaction to see tuples using the snapshot of another transaction
|This would assist multiple backends in working together.
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00400.php <nowiki>Transaction Snapshot Cloning</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00135.php
* http://archives.postgresql.org/pgsql-hackers/2010-12/msg00260.php
* http://archives.postgresql.org/pgsql-hackers/2011-01/msg00466.php
* http://archives.postgresql.org/pgsql-hackers/2011-08/msg00684.php
}}

{{TodoItem
|Consider decreasing the I/O caused by updating tuple hint bits
* [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00847.php <nowiki>Hint Bits and Write I/O</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-07/msg00199.php <nowiki>Re: [HACKERS] Hint Bits and Write I/O</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-10/msg00695.php
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg00792.php
* http://archives.postgresql.org/pgsql-hackers/2011-01/msg01063.php
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01408.php
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01453.php
}}

{{TodoItem
|Avoid the requirement of freezing pages that are infrequently modified
|If all rows on a page are visible, it is possible to set a bit in the visibility map (once the visibility map is 100% reliable) and not need to freeze the page, avoiding a page rewrite
* http://archives.postgresql.org/message-id/4BF701CF.2090205@agliodbs.com
* http://archives.postgresql.org/pgsql-hackers/2010-06/msg00082.php
}}

{{TodoItem
|Avoid reading in b-tree pages when replaying vacuum records in hot standby mode
* [http://archives.postgresql.org/message-id/1272571938.4161.14739.camel@ebony <nowiki>Hot Standby tuning for btree_xlog_vacuum()</nowiki>]
}}

{{TodoItem
|Restructure truncation logic to be more resistant to failure
|This also involves not writing dirty buffers for a truncated or dropped relation
* http://archives.postgresql.org/pgsql-hackers/2010-08/msg01032.php
}}

{{TodoItem
|Consider adding logic to increase large tables by more than 8k
|This would reduce file system fragmentation
* http://archives.postgresql.org/pgsql-bugs/2011-03/msg00337.php
}}

== Miscellaneous Other ==

{{TodoItem
|Deal with encoding issues for filenames in the server filesystem
* {{MessageLink|20090413184335.39BE.52131E4D@oss.ntt.co.jp|a proposed patch here}}
* {{MessageLink|8484.1244655656@sss.pgh.pa.us|some issues about it here}}
* {{MessageLink|20100107103740.97A5.52131E4D@oss.ntt.co.jp|Windows-specific patch here}}
}}

{{TodoItem
|Deal with encoding issues in the output of localeconv()
* [http://archives.postgresql.org/message-id/40c6d9160904210658y590377cfw6dbbecb53d2b8be0@mail.gmail.com bug report]
* [http://archives.postgresql.org/message-id/49EF8DA0.90008@tpf.co.jp draft patch]
* [http://archives.postgresql.org/message-id/21710.1243620986@sss.pgh.pa.us review of patch]
}}

{{TodoItem
|Provide schema name and other fields available from SQL GET DIAGNOSTICS in error reports
* [http://archives.postgresql.org/message-id/dcc563d10810211907n3c59a920ia9eb7cd2a6d5ea58@mail.gmail.com <nowiki>How to get schema name which violates fk constraint</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-11/msg00846.php <nowiki>patch - Report the schema along table name in a referential failure error message</nowiki>]
* {{MessageLink|3191.1263306359@sss.pgh.pa.us|Re: NOT NULL violation and error-message}}
* [http://archives.postgresql.org/pgsql-hackers/2009-08/msg00213.php <nowiki>the case for machine-readable error fields</nowiki>]
}}

{{TodoItemEasy
| Provide [http://developer.postgresql.org/pgdocs/postgres/libpq-connect.html#LIBPQ-CONNECT-FALLBACK-APPLICATION-NAME fallback_application_name] in contrib/pgbench, oid2name, and dblink.
* {{MessageLink|w2g9837222c1004070216u3bc46b3ahbddfdffdbfb46212@mail.gmail.com|fallback_application_name and pgbench}}
}}

{{TodoItem
|Add 64-bit support to /contrib/pgbench
* http://archives.postgresql.org/pgsql-hackers/2010-07/msg00153.php
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg00705.php
}}

== Source Code ==

{{TodoItem
|Add use of 'const' for variables in source tree
* http://archives.postgresql.org/pgsql-hackers/2011-11/msg00473.php
}}

{{TodoItemEasy
|Remove warnings created by -Wcast-align}}

{{TodoItem
|Move platform-specific ps status display info from ps_status.c to ports}}

{{TodoItem
|Add optional CRC checksum to heap and index pages
|One difficulty is how to prevent hint bit changes from affecting the computed CRC checksum.
* http://archives.postgresql.org/message-id/19934.1226601952%40sss.pgh.pa.us
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg00002.php <nowiki>Re: Block-level CRC checks</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg01028.php <nowiki>double-buffering page writes</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-11/msg00524.php <nowiki>Re: Block-level CRC checks</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-12/msg01101.php <nowiki>Re: Block-level CRC checks</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2009-12/msg00011.php <nowiki>Re: Block-level CRC checks</nowiki>]
* http://archives.postgresql.org/pgsql-hackers/2010-11/msg00249.php
* http://archives.postgresql.org/message-id/20111221215913.GA4536@fetter.org
* http://archives.postgresql.org/message-id/CA+U5nMJzQyxcObkpNAf1SYTX-gO_Mom3O9JXHnGpxRo1kXJ7ww@mail.gmail.com
* http://archives.postgresql.org/pgsql-hackers/2012-01/msg00128.php
* http://archives.postgresql.org/pgsql-hackers/2012-01/msg00113.php
* http://archives.postgresql.org/pgsql-hackers/2012-02/msg00172.php
* http://archives.postgresql.org/pgsql-hackers/2012-03/msg00001.php
* http://archives.postgresql.org/pgsql-hackers/2012-03/msg00188.php
}}

{{TodoItem
|Consider a faster CRC32 algorithm
* http://archives.postgresql.org/pgsql-hackers/2010-05/msg01112.php
}}

{{TodoItem
|Allow cross-compiling by generating the zic database on the target system}}

{{TodoItem
|Improve NLS maintenance of libpgport messages linked onto applications}}

{{TodoItem
|Use UTF8 encoding for NLS messages so all server encodings can read them properly}}

{{TodoItem
|Allow creation of universal binaries for Darwin
* [http://archives.postgresql.org/pgsql-hackers/2008-07/msg00884.php <nowiki>Getting to universal binaries for Darwin</nowiki>]
}}

{{TodoItem
|Consider GnuTLS if OpenSSL license becomes a problem
* http://archives.postgresql.org/pgsql-hackers/2011-02/msg00892.php
* [http://archives.postgresql.org/pgsql-patches/2006-05/msg00040.php <nowiki>[PATCH] Add support for GnuTLS</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2006-12/msg01213.php <nowiki>TODO: GNU TLS</nowiki>]
}}

{{TodoItem
|Consider making NAMEDATALEN more configurable in future releases}}

{{TodoItem
|Research use of signals and sleep wake ups
* [http://archives.postgresql.org/pgsql-hackers/2007-07/msg00003.php <nowiki>Restartable signals 'n all that</nowiki>]
}}

{{TodoItem
|Allow C++ code to more easily access backend code
* [http://archives.postgresql.org/pgsql-hackers/2008-12/msg00302.php <nowiki>Mostly Harmless: Welcoming our C++ friends</nowiki>]
}}

{{TodoItem
|Consider simplifying how memory context resets handle child contexts
* [http://archives.postgresql.org/pgsql-patches/2007-08/msg00067.php <nowiki>Re: Memory leak in nodeAgg</nowiki>]
}}

{{TodoItem
|Create three versions of libpgport to simplify client code
* [http://archives.postgresql.org/pgsql-hackers/2007-10/msg00154.php <nowiki>8.4 TODO item: make src/port support libpq and ecpg directly</nowiki>]
}}

{{TodoItem
|Improve detection of shared memory segments being used by others by checking the SysV shared memory field 'nattch'
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00656.php <nowiki>postgresql in FreeBSD jails: proposal</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00673.php <nowiki>Re: postgresql in FreeBSD jails: proposal</nowiki>]
}}

{{TodoItem
|Implement the non-threaded Avahi service discovery protocol
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00939.php <nowiki>Re: [PATCHES] Avahi support for Postgresql</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-02/msg00097.php <nowiki>Re: Avahi support for Postgresql</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg01211.php <nowiki>Re: [PATCHES] Avahi support for Postgresql</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-04/msg00001.php <nowiki>Re: [HACKERS] Avahi support for Postgresql</nowiki>]
}}

{{TodoItem
|Reduce data row alignment requirements on some 64-bit systems
* [http://archives.postgresql.org/pgsql-hackers/2008-10/msg00369.php <nowiki>[WIP] Reduce alignment requirements on 64-bit systems.</nowiki>]
}}

{{TodoItem
|Restructure TOAST internal storage format for greater flexibility
* [http://archives.postgresql.org/pgsql-hackers/2008-11/msg00049.php <nowiki>Re: PG_PAGE_LAYOUT_VERSION 5 - time for change</nowiki>]
}}

{{TodoItem
| Add regression tests for pg_dump/restore
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01967.php <nowiki>"make install-check-pg_dump" target in src/regress]</nowiki>]
}}

{{TodoItem
| Research different memory allocation methods for lists
* http://archives.postgresql.org/pgsql-hackers/2011-04/msg01467.php
}}

{{TodoItem
| Consider removing the attribute options cache
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg00039.php
}}

{{TodoItem
| Restructure /contrib section
* http://archives.postgresql.org/pgsql-hackers/2011-06/msg00705.php
}}

=== /contrib/pg_upgrade ===
{{TodoSubsection}}

{{TodoItem
|Handle large object comments
|This is difficult to do because the large object doesn't exist when --schema-only is loaded.
}}

{{TodoItem
|Consider using pg_depend for checking object usage in version.c
}}

{{TodoItem
|If reindex is necessary, allow it to be done in parallel with pg_dump custom format
}}

{{TodoItem
|Migrate pg_statistic by dumping it out as a flat file, so analyze is not necessary
|pg_class.oid is not preserved so schema.tablename must be used.
* [http://archives.postgresql.org/message-id/CAAZKuFaWdLkK8eozSAooZBets9y_mfo2HS6urPAKXEPbd-JLCA@mail.gmail.com pg_upgrade and statistics]
}}

{{TodoItem
|Improve testing, perhaps using the buildfarm
|The buildfarm has access to multiple versions of PostgreSQL.
}}

{{TodoItem
|Create machine-readable output of pg_controldata
|This would avoid parsing its output. The problem is we need pg_controldata output from both the old and new clusters so we would need to support both formats.
}}

{{TodoEndSubsection}}

=== Windows ===
{{TodoSubsection}}

{{TodoItem
|Remove configure.in check for link failure when cause is found}}

{{TodoItem
|Remove readdir() errno patch when runtime/mingwex/dirent.c rev 1.4 is released}}

{{TodoItem
|Allow psql to use readline once non-US code pages work with backslashes}}

{{TodoItem
|Fix problem with shared memory on the Win32 Terminal Server}}

{{TodoItem
|Improve signal handling
* [http://archives.postgresql.org/pgsql-patches/2005-06/msg00027.php <nowiki>Simplify Win32 Signaling code</nowiki>]
}}

{{TodoItem
|Convert MSVC build system to remove most batch files
* [http://archives.postgresql.org/pgsql-hackers/2007-08/msg00961.php <nowiki>MSVC build system</nowiki>]
}}

{{TodoItem
|Support pgxs when using MSVC}}

{{TodoItem
|Fix MSVC NLS support, like for to_char()
* [http://archives.postgresql.org/pgsql-hackers/2008-02/msg00485.php <nowiki>NLS on MSVC strikes back!</nowiki>]
* [http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php <nowiki>Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)</nowiki>]
}}

{{TodoItem
|Find a correct rint() substitute on Windows
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00808.php <nowiki>Minor bug in src/port/rint.c</nowiki>]
}}

{{TodoItem
|Fix global namespace issues when using multiple terminal server sessions
* [http://archives.postgresql.org/message-id/48F3BFCC.8030107@dunslane.net problems with Windows global namespace]}}

{{TodoItem
|Change from the current autoconf/gmake build system to cmake
* [http://archives.postgresql.org/pgsql-hackers/2008-12/msg01869.php <nowiki>About CMake (was Re: [COMMITTERS] pgsql: Append major version number and for libraries soname major)</nowiki>]
}}

{{TodoItem
|Improve consistency of path separator usage
* http://archives.postgresql.org/message-id/49C0BDC5.4010002@hagander.net
}}

{{TodoItem
|Fix cross-compiling on Windows
* http://archives.postgresql.org/pgsql-bugs/2010-10/msg00110.php
}}

{{TodoItemDone
|Allow multiple Postgres clusters running on the same machine to distinguish themselves in the event log
* http://archives.postgresql.org/pgsql-hackers/2011-03/msg01297.php
* http://archives.postgresql.org/pgsql-hackers/2011-05/msg00574.php
}}

{{TodoEndSubsection}}

=== Wire Protocol Changes ===
{{TodoSubsection}}

{{TodoItem
|Allow dynamic character set handling}}

{{TodoItem
|Add decoded type, length, precision}}

{{TodoItem
|Mark result columns as known-not-null when possible
* [http://archives.postgresql.org/pgsql-hackers/2010-11/msg01029.php <nowiki>Adding nullable indicator to Describe</nowiki>]
}}

{{TodoItem
|Provide more control over planner treatment of statements being prepared}}

{{TodoItem
|Use compression?}}

{{TodoItem
|Update clients to use data types, typmod, schema.table.column names of result sets using new statement protocol}}

{{TodoItem
|Set protocol for wire format negotiation
* [http://archives.postgresql.org/message-id/CACMqXCKkGrGXxQhjHCKCe0B8hn6sTt-1sdgHZOSGQMxrusOsQA@mail.gmail.com GUC_REPORT for protocol tunables]
}}

{{TodoItem
|Make sure upgrading to a 4.1 protocol version will actually work smoothly
* [http://archives.postgresql.org/message-id/28307.1318255008@sss.pgh.pa.us Re: libpq, PQdescribePrepared -> PQftype, PQfmod, no PQnullable]
}}

{{TodoEndSubsection}}

== Documentation ==

{{TodoItem
|Convert single quotes to apostrophes in the PDF documentation
* [http://archives.postgresql.org/pgsql-docs/2007-12/msg00059.php <nowiki>SGML docs and pdf single-quotes</nowiki>]
}}

{{TodoItem
|Provide a manpage for postgresql.conf
* {{messageLink|20080819194311.GH4428@alvh.no-ip.org|A smaller default postgresql.conf}}
* {{messageLink|200808211910.37524.peter_e@gmx.net|A smaller default postgresql.conf}}
}}

{{TodoItem
|Change the manpage-generating toolchain to use the new XML-based docbook2x tools
* {{messageLink|200808211910.37524.peter_e@gmx.net|A smaller default postgresql.conf}}
}}

{{TodoItem
|Consider changing documentation format from SGML to XML
* [http://archives.postgresql.org/pgsql-docs/2006-12/msg00152.php <nowiki>Re: Authoring Tools WAS: Switching to XML</nowiki>]
* http://archives.postgresql.org/pgsql-docs/2011-04/msg00020.php
* http://wiki.postgresql.org/wiki/Switching_PostgreSQL_documentation_from_SGML_to_XML
}}

{{TodoItem
|Document support for N<nowiki>' '</nowiki> national character string literals, if it matches the SQL standard
* http://archives.postgresql.org/message-id/1275895438.1849.1.camel@fsopti579.F-Secure.com
}}

{{TodoItem
|Add diagrams to the documentation
* http://archives.postgresql.org/pgsql-docs/2010-07/msg00001.php
}}

== Exotic Features ==

{{TodoItem
|Add pre-parsing phase that converts non-ISO syntax to supported syntax
|This could allow SQL written for other databases to run without modification.}}

{{TodoItem
|Allow plug-in modules to emulate features from other databases}}

{{TodoItem
|Add features of Oracle-style packages
|A package would be a schema with session-local variables, public/private functions, and initialization functions. It is also possible to implement these capabilities in any schema and not use a separate "packages" syntax at all.
* [http://archives.postgresql.org/pgsql-hackers/2006-08/msg00384.php <nowiki>proposal for PL packages for 8.3.</nowiki>]
}}

{{TodoItem
|Consider allowing control of upper/lower case folding of unquoted identifiers
* [http://archives.postgresql.org/pgsql-hackers/2004-04/msg00818.php <nowiki>Bringing PostgreSQL torwards the standard regarding case folding</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2006-10/msg01527.php <nowiki>Re: [SQL] Case Preservation disregarding case sensitivity?</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-03/msg00849.php <nowiki>TODO Item: Consider allowing control of upper/lower case folding of unquoted, identifiers</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-07/msg00415.php <nowiki>Identifier case folding notes</nowiki>]
* [http://archives.postgresql.org/pgsql-hackers/2008-07/msg00415.php <nowiki>Identifier case folding notes</nowiki>]
}}

{{TodoItem
|Add autonomous transactions
* [http://archives.postgresql.org/pgsql-hackers/2008-01/msg00893.php <nowiki>autonomous transactions</nowiki>]
}}

{{TodoItem
|Give query progress indication
* [[Query progress indication]]
}}

{{TodoItem
|Rethink our type system
* [[Rethinking datatypes]]
}}

== Features We Do ''Not'' Want ==

The following features have been discussed ad nauseum on the PostgreSQL mailing lists and the consensus has been that the project is not interested in them. As such, if you are going to bring them up as potential features, you will want to be familiar with all of the arguments against these features which have been previously made over the years. If you decide to work on such features anyway, you should be aware that you face a higher-than-normal barrier to get the Project to accept them.

{{TodoItem
|All backends running as threads in a single process (not wanted)
|This eliminates the process protection we get from the current setup. Thread creation is usually the same overhead as process creation on modern systems, so it seems unwise to use a pure threaded model, and MySQL and DB2 have demonstrated that threads introduce as many issues as they solve. Threading specific operations such as I/O, seq scans, and connection management has been discussed and will probably be implemented to enable specific performance features. Moving to a threaded engine would also require halting all other work on PostgreSQL for one to two years.}}

{{TodoItem
|"Oracle-style" optimizer hints (not wanted)
|Optimizer hints, as implemented in Oracle and other RDBMSes, are used to work around problems in the optimizer and introduce upgrade and maintenance issues. We would rather have such problems reported and fixed. We have discussed a more sophisticated system of per-class cost adjustment instead, but a specification remains to be developed. See [[OptimizerHintsDiscussion|Optimizer Hints Discussion]] for further information.}}

{{TodoItem
|Embedded server (not wanted)
|While PostgreSQL clients runs fine in limited-resource environments, the server requires multiple processes and a stable pool of resources to run reliably and efficiently. Stripping down the PostgreSQL server to run in the same process address space as the client application would add too much complexity and failure cases. Besides, there are several very mature embedded SQL databases already available.}}

{{TodoItem
|Obfuscated function source code (not wanted)
|Obfuscating function source code has minimal protective benefits because anyone with super-user access can find a way to view the code. At the same time, it would greatly complicate backups and other administrative tasks. To prevent non-super-users from viewing function source code, remove SELECT permission on pg_proc.
* [http://archives.postgresql.org/pgsql-general/2008-09/msg00668.php <nowiki>Obfuscated stored procedures (was Re: Oracle and Postgresql)</nowiki>]
}}

{{TodoItem
|Indeterminate behavior for the GROUP BY clause (not wanted)
|At least one other database product allows specification of a subset of the result columns which GROUP BY would need to be able to provide predictable results; the server is free to return any value from the group. This is not viewed as a desirable feature. PostgreSQL 9.1 allows result columns that are not referenced by GROUP BY if a primary key for the same table is referenced in GROUP BY.
* [http://archives.postgresql.org/pgsql-hackers/2010-03/msg00297.php <nowiki>Re: SQL compatibility reminder: MySQL vs PostgreSQL</nowiki>]
}}

</div>

[[Category:Todo]]

Todo

Kgrittn — Tue, 20 Mar 2012 16:26:14 GMT

Kgrittn: /* Locking */ Mark SSI optimization item done which was committed to 9.2 and a 9.1.1.

PgCon 2012 Developer Meeting

Kgrittn — Fri, 02 Mar 2012 17:58:48 GMT

Kgrittn: /* Proposed Agenda Items */ Include Kevin for Queuing agenda item, and remove question mark on Kevin for Materialized views

User:Kgrittn

Kgrittn — Wed, 29 Feb 2012 17:26:04 GMT

Kgrittn: /* Serializable transaction isolation level */

Information about Kevin Grittner and his activities.

[[image:Kevin-Grittner.jpg]]

== Bio ==

Kevin is currently employed as a database administrator with the [http://www.wicourts.gov/about/organization/offices/ccap.htm Consolidated Court Automation Programs (CCAP)] at the [http://www.wicourts.gov Wisconsin Court System]. He's been making a living in the computer industry since 1972. He started a consulting company in 1980 after working in data control, operations, programming, systems analysis, and design. He was finally lured back to employee status in his current position after more than a quarter century of consulting.

Kevin has experience with many types of applications in many types of organizations, businesses, and government. In 1984 he was the architect and primary author of the PROBER database and development system, used by thousands of corrections agencies, health departments, hospitals, and fire departments around the world. (The last known use was a statewide health department which converted off the product in 2006.) He has also developed application development platforms used by clients to speed development, improve standardization, and provide portability.

== Current Work In Process ==

=== Declarative materialized views ===

For the 9.3 release.

== Possible Future Work ==

=== Rewrite tsearch parser to use regular expressions ===

In reviewing a patch to fix some performance problems in the current parser, I became interested in the possibility of rewriting the current state machine implementation with a regular expression implementation.

[http://archives.postgresql.org/message-id/200912102005.16560.andres@anarazel.de Re: tsearch parser inefficiency if text includes urls or emails - new version]

[http://archives.postgresql.org/message-id/4B210D9E020000250002D344@gw.wicourts.gov tsearch parser overhaul]

=== Deleted WAL files held open by backends in Linux ===

I wasted some time tracking down an oddity which is more of an annoyance in system administration than a real problem, but might look at cleaning it up to save others the bother of investigating the issue.

[http://archives.postgresql.org/message-id/15412.1259630304@sss.pgh.pa.us Deleted WAL files held open by backends in Linux]

=== Temporal data improvements ===

Temporal data handling is weak in SQL in general, and the PostgreSQL implementation seems skewed toward scientific or engineering applications, leaving it weaker than some databases on business applications. (One example would be clean handling of monthy payment schedules.) Enhancements to this area might be interesting.

[http://archives.postgresql.org/message-id/4AE944BD.90809@comcast.net Proposal - temporal contrib module]

[http://archives.postgresql.org/message-id/48692c2d0911171231h6ab16a64yc4db35a6e26909e0@mail.gmail.com Re: Timezones (in 8.5?)]

=== LSB script compliance ===

I came up with a pretty LSB compliant script for Linux; however, the community feels that most of the logic handled in the shell in this script should be moved into pg_ctl. There's a pretty long and winding thread on the topic. The last version of the script might be useful to identify what issues need to be covered in the current pg_ctl code.

[http://archives.postgresql.org/message-id/4A8C41EC.3080708@agliodbs.com We should Axe /contrib/start-scripts]

[http://archives.postgresql.org/message-id/4A9581A5020000250002A37A@gw.wicourts.gov Linux LSB init script]

=== TOAST improvements ===

There have been a few posts to the lists about specific use cases where TOAST defaults are far from optimal. Allowing some tuning here might be helpful.

[http://archives.postgresql.org/message-id/4A6088D50200002500028940@gw.wicourts.gov Higher TOAST compression]

=== Literal and NULL handling anomalies ===

There are a few corner cases where differences between standard and PostgreSQL behaviors with string literals or NULLs astonish those new to PostgreSQL. These aren't easily solved, but might be worth the effort.

=== README files ===

Language is not always as readable as it could be, and some files still largely read like proposals for features which are now implemented.

Replication, Clustering, and Connection Pooling

Kgrittn — Mon, 19 Dec 2011 17:58:04 GMT

Kgrittn: Change obsolete references to http://pgpool.projects.postgresql.org/ to new location at http://www.pgpool.net/

==Introduction==
There are many approaches available to scale PostgreSQL beyond running on a single server. An outline of the terminology and basic technologies involved is at [http://www.postgresql.org/docs/current/interactive/high-availability.html High Availability and Load Balancing]. There is a [http://momjian.us/main/writings/pgsql/replication.pdf presentation] covering some of these solutions.

There is no one-size fits all replication software. You have to understand your requirements and how various approaches fit into that. For example, here are two extremes in the replication problem space:

* You have a few servers connected to a local network you want to always keep current for failover and load-balancing purposes. Here you would be considering solutions that are synchronous, eager, and therefore conflict-free.
* Your users take a local copy of the database with them on laptops when they leave the office, make changes while they are away, and need to merge those with the main database when they return. Here you'd want an asynchronous, lazy replication approach, and will be forced to consider how to handle conflicts in cases where the same record has been modified both on the master server and on a local copy.

These are both database replication problems, but the best way to solve them is very different. And as you can see from these examples, replication has a lot of specific terminology that you'll have to understand to figure out what class of solution makes sense for your requirements. A great source for this background is in the
[http://www.postgres-r.org/documentation/terms Postgres-R Terms and Definitions for Database Replication]. The main theoretical topic it doesn't mention is how to resolve conflict resolution in lazy replication cases like the laptop situation, which involves voting and similar schemes.

==Features in the Core of PostgreSQL==
The PostgreSQL core team considered replication and clustering technology outside the scope of the main project's focus but this changed in Spring 2008, see the [http://archives.postgresql.org/pgsql-hackers/2008-05/msg00913.php Core Team's statement].

*[[Hot Standby]]/[[Streaming Replication]] is available as of PostgreSQL 9.0 and provides asynchronous binary replication to one or more standbys. Standbys may also become hot standbys meaning they can be queried as a read-only database. This is the fastest type of replication available as WAL data is sent immediately rather than waiting for a whole segment to be produced and shipped.

*[[Warm Standby]]/Log Shipping is a HA solution which 'replicates' a database cluster to an archive or a warm (can be brought up quickly, but not available for querying) standby server. Overhead is very low and it's easy to set up. This is a simple and appropriate solution if all you care about is continuous backup and short failover times.

==Comparison matrix==

This page is being overhauled at [[Clustering]]

{| border="1" cellpadding="1" cellspacing="0" style="font-size: 85%; border: gray solid 1px; border-collapse: collapse; text-align: center; width: 100%; table-layout: fixed;"
|- style="background: #ececec"
! Program
! License
! Maturity
! Replication Method
! Sync
! Connection Pooling
! Load Balancing
! Query Partitioning
|-
! style="text-align:block;" bgcolor="#ececec" | [http://pgcluster.projects.postgresql.org/ PGCluster]
| bgcolor="#ffffaa" | BSD
| bgcolor="#ffffaa" | See version details on site
| bgcolor="#ffffaa" | Master-Master
| bgcolor="#ffffaa" | Synchronous
| bgcolor="#ffaaaa" | No
| bgcolor="#ddffdd" | Yes
| bgcolor="#ffaaaa" | No
|-
! style="text-align:block;" bgcolor="#ececec" | '''pgpool-I'''
| bgcolor="#ffffaa" | BSD
| bgcolor="#ffffaa" | Stable
| bgcolor="#ffffaa" | Statement-Based Middleware
| bgcolor="#ffffaa" | Synchronous
| bgcolor="#ddffdd" | Yes
| bgcolor="#ddffdd" | Yes
| bgcolor="#ffaaaa" | No
|-
! style="text-align:block;" bgcolor="#ececec" | [http://www.pgpool.net/ pgpool-II]
| bgcolor="#ffffaa" | BSD
| bgcolor="#ffffaa" | Recent release
| bgcolor="#ffffaa" | Statement-Based Middleware
| bgcolor="#ffffaa" | Synchronous
| bgcolor="#ddffdd" | Yes
| bgcolor="#ddffdd" | Yes
| bgcolor="#ddffdd" | Yes
|-
! style="text-align:block;" bgcolor="#ececec" | [http://slony.info/ slony-I]
| bgcolor="#ffffaa" | BSD
| bgcolor="#ffffaa" | Stable
| bgcolor="#ffffaa" | Master-Slave
| bgcolor="#ffffaa" | Asynchronous
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
|-
! style="text-align:block;" bgcolor="#ececec" | [http://bucardo.org/ Bucardo]
| bgcolor="#ffffaa" | BSD
| bgcolor="#ffffaa" | Stable
| bgcolor="#ffffaa" | Master-Master, Master-Slave
| bgcolor="#ffffaa" | Asynchronous
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
|-
! style="text-align:block;" bgcolor="#ececec" | [http://wiki.postgresql.org/wiki/Skytools Londiste]
| bgcolor="#ffffaa" | BSD
| bgcolor="#ffffaa" | Stable
| bgcolor="#ffffaa" | Master-Slave
| bgcolor="#ffffaa" | Asynchronous
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
|-
! style="text-align:block;" bgcolor="#ececec" | [http://www.commandprompt.com/products/mammothreplicator/ Mammoth]
| bgcolor="#ffffaa" | BSD
| bgcolor="#ffffaa" | Stable
| bgcolor="#ffffaa" | Master-Slave
| bgcolor="#ffffaa" | Asynchronous
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
|-
! style="text-align:block;" bgcolor="#ececec" | [http://www.rubyrep.org/ rubyrep]
| bgcolor="#ffffaa" | MIT
| bgcolor="#ffffaa" | Recent Release
| bgcolor="#ffffaa" | Master-Master, Master-Slave
| bgcolor="#ffffaa" | Asynchronous
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
| bgcolor="#ffaaaa" | No
|}

==Replication==

Aside from Warm Standby, mentioned above...

*Slony-I: Seems good, single master only, master is a single point of failure, no good failover system for electing a new master or having a failed master rejoin the cluster. Slave databases are mostly for safety or for parallelizing queries for performance. Suffers from O(N^2) communications (N = cluster size). with reasonable sysadmin you can implement failover system yourself. regarding communications, you can cascade the replication to reduce load on the master. If you were implementing a large replication cluster, this would probably be a good idea. Slony is powerful, trigger based, and highly configurable.

* PGCluster: PGCluster (which, incidentally, is not the same as PGCluster-II, a shared-disk solution), which does synchronous multimaster replication. Two single-points failure spots, load balancer and the data replicator. The project has historically looked a bit dead, but they just released a new version and http://pgfoundry.org/projects/pgcluster is up to date (at least downloads page) One major downside to PGCluster is that it uses a modified version of PostgreSQL, and it usually lags a few releases behind.

* http://www.pgpool.net/ pgpool 1/2 is a reasonable solution. it's statement level replication, which has some downsides, but is good for certain things. pgpool 2 has a neat distributed table mechanism which is interesting. You might want to be looking here if you have extremely high ratios of read to write but need to service a huge transaction volume. Supports load-balancing and replication by implementing a proxy that duplicates all updates to all slaves. It can partition data by doing this, and it can semi-intelligently route queries to the appropriate servers.

* "Mammoth Replicator" - BSD - http://www.commandprompt.com/products/mammothreplicator/ - Former proprietary solution, now open source. Uses a central logging process to distribute data changes amongst nodes. Essentially a fork of Postgres, as the changes are written directly into the backend.

* "Bucardo" - BSD License - http://bucardo.org/ - Trigger-based, asynchronous, multi-master or master-slave, written using plperl.

* Cybertec, an Austrian company, offers a proprietary packaging of PGCluster. They simply call it PostgreSQL Multimaster-Replication, see http://www.cybertec.at.

* [[Londiste_Tutorial|Londiste]], a part of [[Skytools]] (https://developer.skype.com/SkypeGarage/DbProjects/SkyTools) which is a collection of replication tools from the Skype people. Purports to be simpler to use than Slony.

* [http://www.continuent.com/index.php?option=com_content&task=view&id=212&Itemid=169 Continuent uni/cluster], proprietary and the related Sequoia (jdbc, formerly known as c-jdbc)

* [http://www.postgres-r.org Postgres-R] is still in development. It features eager and thus conflict-free, but async multi-master replication.

* [http://symmetricds.codehaus.org/ SymmetricDS] is an open-source, web-enabled, database independent, data synchronization software application. It uses web and database technologies to replicate tables between relational databases in near real time. The software was designed to scale for a large number of databases, work across low-bandwidth connections, and withstand periods of network outages. Supports several relational databases, including PostgreSQL. Licensed under Lesser GPL (LGPL).

* DRBD (http://www.drbd.org/), a device driver that replicates disk blocks to other nodes. This works for failover only, not for scaling reads. Easy migration of devices if combined with an NFS export.

* [http://sourceforge.net/projects/daffodilreplica/ Daffodil Replicator]. Supports several relational databases, including PostgreSQL. Licensed under GPL.

* "RubyRep" - MIT License - http://www.rubyrep.org/ - Ruby based, asynchronous, multi-master replication system, which supports Postgres and MySQL.

* "pg_comparator" - BSD License - http://pgfoundry.org/projects/pg-comparator/ - Perl-based, table-level async master-slave "diff" and "patch" method of replication. Low configuration overhead.

===Inactive projects===
* Slony-II
* PGReplication

==Clustering==

* [http://www.greenplum.com/index.php?page=greenplum-database Greenplum Database] (formerly Bizgres MPP), proprietary. Not so much a replication solution as a way to parallelize queries, and targeted at the data warehousing crowd. Similar to ExtenDB, but tightly integrated with PostgreSQL.

*[http://www.enterprisedb.com/products/gridsql.do GridSQL for EnterpriseDB Advanced Server] (formerly ExtenDB)

*sequoia (jdbc, formerly known as c-jdbc)

* [[PL/Proxy]] - database partitioning system implemented as PL language.

*[http://db.cs.yale.edu/hadoopdb/hadoopdb.html HadoopDB] - A MapReduce layer put in front of a cluster of postgres back end servers. Shared-nothing clustering.

==Connection Pooling and Acceleration==

Connection pooling programs let you reduce database-related overhead when it's the sheer number of physical connections dragging performance down. This is particularly important on Windows, where system limitations prevent large number of connections; see "I cannot run with more than about 125 connections at once" in the [http://www.postgresql.org/docs/faqs.FAQ_windows.html Windows FAQ]. It's also vital for web applications where the number of connections can get very large.

Some programs that implement connection pooling are:
* [[PgBouncer]]
* [http://www.pgpool.net/ pgpool]

Some people also or alternately use [http://www.danga.com/memcached/ memcached] in various ways to reduce the work the database handles directly by caching popular data.

==Credits==

Sources for the initial information on this page include:
*[http://archives.postgresql.org/pgsql-performance/2007-06/msg00264.php replication thread]
*[http://archives.postgresql.org/pgsql-general/2007-08/msg00085.php pgpool2 vs sequoia]
*[http://archives.postgresql.org/pgsql-hackers/2006-10/msg00810.php Postgresql Caching]

A existing page covering this topic in German is at http://burger-ag.de/postgresql_replikation.whtml It translates pretty well through [http://babelfish.altavista.com/ Babelfish].

Sources for more information located but not yet integrated into here:
* [http://bristlecone.continuent.org/uploads/bristlecone/HomePage/PG_East-Scale-Out-Benchmarks_FINAL2.pdf Portable Scale-Out Benchmarks for PostgreSQL] by Robert Hodges
* [http://www.fastware.com.au/docs/PostgreSQL_HighAvailability.pdf High Availability and PostgreSQL] by Gavin Sherry

[[Category:Replication]][[Category:Administration]][[Category:Performance]][[Category:Clustering]]

SSI/fr

Kgrittn — Sun, 27 Nov 2011 00:07:43 GMT

Kgrittn: /* Rapport de Dépôt */ Correct typo.

{{Languages}}

Documentation de la «Serializable Snapshot Isolation» (Isolation par Instantanés Sérialisables, ou SSI) dans PostgreSQL, comparée à la «Snapshot Isolation» (Isolation par Instantanés, ou SI). Celles-ci correspondent respectivement aux niveaux d'isolation de transaction SERIALIZABLE et REPEATABLE READ dans PostgreSQL, à partir de la version 9.1.

== Aperçu ==

Avec de vraies transactions sérialisables, si vous pouvez prouver que votre transaction fera ce qui est prévu si il n'y a aucune transaction concurrente, elle fera ce qui est prévu quelles que soient les autres transactions sérialisables qui s'exécuteront en même temps qu'elle, ou sera annulée pour erreur de sérialisation.

Ce document montre les problèmes qui peuvent se produire avec certaines combinaisons de transactions au niveau d'isolation de transaction REPEATABLE READ, et comment elles sont évitées avec le niveau d'isolation SERIALIZABLE, à partir de PostgreSQL 9.1.

Ce document est destiné au programmeur d'applications ou à l'administrateur de bases de données. Pour les détails sur l'implémentation de SSI, voyez la page de Wiki [[Serializable]]. Pour plus d'informations sur comment utiliser ce niveau d'isolation, voyez [http://docs.postgresql.fr/current/transaction-iso.html#XACT-SERIALIZABLE la documentation PostgreSQL courante].

== Exemples ==

Dans les environnements qui évitent de protéger leur intégrité en mettant en place des verrous bloquants, il sera fréquent que la base soit configurée (dans postgresql.conf) avec:
default_transaction_isolation = 'serializable'
Pour cette raison, tous les exemples ont été effectués avec ce paramétrage, ce qui a évité de polluer les exemples en se contentant d'un simple begin plutôt que de déclarer explicitement le niveau d'isolation pour chaque transaction.

=== Write Skew Simple (Écriture Faussée Simple?) ===

Quand deux transactions concurrentes déterminent chacune ce qu'elles écrivent en lisant des données qui se chevauchent avec des données que l'autre modifie, on peut se retrouver dans un état qui ne devrait pas apparaître si une des deux s'était exécutée avant l'autre. C'est un phénomène connu sous le nom de ''write skew'', et c'est la forme la plus simple de défaut de sérialisation contre laquelle SSI vous protège.

Quand il y a write skew dans SSI, les deux transactions se déroulent jusqu'à ce que l'une valide. La première à valider gagne, et l'autre transaction est annulée. La règle du "le premier à valider gagne" garantit que du travail peut avoir lieu sur la base et que la transaction qui est annulée puisse être tentée à nouveau immédiatement.

----
==== Noir et Blanc ====

Dans ce cas, il y a des enregistrement avec une colonne couleur contenant 'blanc' ou 'noir'. Deux utilisateurs essayent simultanément de convertir tous les enregistrements vers une couleur unique, mais chacun dans une direction opposée. Un veut tout passer tous les blancs en noir, et l'autre tous les noirs en blanc.

L'exemple peut être mis en place avec ces ordres:
create table points
(
id int not null primary key,
couleur text not null
);
insert into points
with x(id) as (select generate_series(1,10))
select id, case when id % 2 = 1 then 'noir'
else 'blanc' end from x;
{|
|+ Exemple Noir et Blanc
! session 1
! session 2
|-
|
begin;
update points set couleur = 'noir'
where couleur = 'blanc';
|-
| ||
begin;
update points set couleur = 'blanc'
where couleur = 'noir';
À ce moment, une des deux transaction est condamnée à mourir.
commit;
Le premier à valider gagne.
select * from points order by id;

id | couleur
----+-------
1 | blanc
2 | blanc
3 | blanc
4 | blanc
5 | blanc
6 | blanc
7 | blanc
8 | blanc
9 | blanc
10 | blanc
(10 rows)
Celle-ci s'est exécutée comme si elle était seule.
|-
|
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
Une erreur de sérialisation. On annule et on réessaye.
rollback;
begin;
update points set couleur = 'noir'
where couleur = 'blanc';
commit;
Il n'y a pas de transaction concurrente pour gêner.
select * from points order by id;

id | couleur
----+-------
1 | noir
2 | noir
3 | noir
4 | noir
5 | noir
6 | noir
7 | noir
8 | noir
9 | noir
10 | noir
(10 rows)
La transaction s'est exécutée seule, après l'autre.
|}

----
==== Données en intersection ====

Cet exemple est tiré de la documentation PostgreSQL. Deux transactions concurrentes lisent des données, et chacune utilise ces données pour mettre à jour l'ensemble lu par l'autre. Un exemple simple, même si un peu artificiel, de données faussées.

L'exemple peut être mis en place avec ces ordres:
CREATE TABLE mytab
(
class int NOT NULL,
value int NOT NULL
);
INSERT INTO mytab VALUES
(1, 10), (1, 20), (2, 100), (2, 200);
{|
|+ Exemple de données en intersection
! session 1
! session 2
|-
|
BEGIN;
SELECT SUM(value) FROM mytab WHERE class = 1;

sum
-----
30
(1 row)

INSERT INTO mytab VALUES (2, 30);
|-
| ||
BEGIN;
SELECT SUM(value) FROM mytab WHERE class = 2;

sum
-----
300
(1 row)

INSERT INTO mytab VALUES (1, 300);
Chaque transaction a modifié ce que l'autre transaction aurait lu. Si les deux étaient autorisées à valider, le comportement sérialisable ne serait plus respecté, parce que si elles avaient été exécutées une seule à la fois, une des transactions aurait vu l'INSERT que l'autre a validé. Nous attendons qu'une des transactions ait validé avant d'annuler quoi que ce soit, toutefois, pour garantir que des traitements soient effectués et éviter que le système ne s'effondre.
COMMIT;
|-
|
COMMIT;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
Donc, maintenant nous annulons la transaction en échec et nous la réessayons depuis le début.
ROLLBACK;
BEGIN;
SELECT SUM(value) FROM mytab WHERE class = 1;

sum
-----
330
(1 row)

INSERT INTO mytab VALUES (2, 330);
COMMIT;
Cela réussit, et le résultat est cohérent avec une exécution sérialisée des transactions.
SELECT * FROM mytab;

class | value
-------+-------
1 | 10
1 | 20
2 | 100
2 | 200
1 | 300
2 | 330
(6 rows)
|}

----
==== Protection contre le Découvert ====

Le cas hypothétique est celui d'une banque qui autorise ses clients à retirer de l'argent jusqu'au total de tout ce qu'ils ont sur tous leurs comptes. La banque transfèrera ensuite automatiquement les fonds au besoin pour terminer la journée avec un solde positif sur chaque compte. À l'intérieur d'une seule transaction, on vérifie que la somme de tous les comptes dépasse la somme requise.

Quelqu'un essaye d'être malin et de piéger la banque en soumettant deux retraits de 900$ sur deux comptes ayant chacun 500$ de solde simultanément. Au niveau d'isolation de transaction REPEATABLE READ, cela pourrait marcher; mais si le niveau d'isolation de transaction SERIALIZABLE est utilisé, SSI détectera une "structure dangereuse" dans le schéma de lecture/écriture et rejettera une des deux transactions.

Cet exemple peut être mis en place avec ces ordres:

create table compte
(
nom text not null,
type text not null,
solde money not null default '0.00'::money,
primary key (nom, type)
);
insert into compte values
('kevin','epargne', 500),
('kevin','courant', 500);

{|
|+ Exemple de Protection contre le Découvert
! session 1
! session 2
|-
|
begin;
select type, solde from compte
where nom = 'kevin';

type | solde
-----------+---------
epargne | $500.00
courant | $500.00
(2 rows)
Le total est de $1000, un retrait de $900 est donc permis.
|-
| ||
begin;
select type, solde from compte
where nom = 'kevin';

type | solde
-----------+---------
epargne | $500.00
courant | $500.00
(2 rows)
Le total est de $1000, un retrait de $900 est donc permis.
|-
|
update compte
set solde = solde - 900::money
where nom = 'kevin' and type = 'epargne';
Jusqu'ici tout va bien.
|-
| ||
update compte
set solde = solde - 900::money
where nom = 'kevin' and type = 'courant';
Maintenant nous avons un problème. Cela ne peut co-exister avec l'activité de l'autre transaction. Nous n'annulons pas encore, parce que la transaction échouerait avec les mêmes conflits si on la réessayait. Le premier à valider va gagner, et l'autre échouera quand elle essayera de continuer après cela.
|-
|
commit;
Celle ci a validé la première. Son travail est enregistré.
|-
| ||
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
Cette transaction n'a pas réussi à retirer l'argent.
Maintenant nous l'annulons et réessayons la transaction.

rollback;
begin;
select type, solde from compte
where nom = 'kevin';

type | solde
-----------+----------
epargne | -$400.00
courant | $500.00
(2 rows)
On voit qu'il y a un solde net de $100. Cette demande de $900 sera rejetée par l'application.
|}

=== Trois Transactions ou Plus ===

Des anomalies de sérialisation peuvent résulter de motifs plus complexes d'accès, impliquant trois transactions ou plus.

----
==== Couleurs Primaires ====

C'est similaire à l'exemple "Blanc et Noir" précédent, à la différence que nous utilisons les trois couleurs primaires. Une transaction essaye de passer le rouge à jaune, la suivante le jaune au bleu, et la troisième le bleu au rouge. Si ces transactions étaient exécutées une seule à la fois, on aurait à la fin de l'exécution deux des trois couleurs, en fonction de l'ordre d'exécution. Si deux d'entre elles sont exécutées simultanément, celle essayant de lire les enregistrements mis à jour par l'autre semblera s'exécuter première, puisqu'elle ne verra pas le travail de l'autre transaction, il n'y a donc pas de problème dans ce cas. Que l'autre transaction soit exécutée avant ou après cela, les résultats sont cohérents avec un ordre d'exécution sérialisé.

Si les trois s'exécutent en même temps, il y a un cycle dans l'ordre apparent d'exécution. Une transaction Repeatable Read ne détecterait pas cela, et la table aurait toujours trois couleurs. Une transaction Sérialisable détectera le problème et annulera une des transactions avec une erreur de sérialisation.

L'exemple peut être mis en place avec ces ordres:
create table points
(
id int not null primary key,
couleur text not null
);
insert into points
with x(id) as (select generate_series(1,9000))
select id, case when id % 3 = 1 then 'rouge'
when id % 3 = 2 then 'jaune'
else 'blue' end from x;
create index points_couleur on points (couleur);
analyze points;
{|
|+ Primary Colors Example
! session 1
! session 2
! session 3
|-
|
begin;
update points set couleur = 'jaune'
where couleur = 'rouge';
|-
| ||
begin;
update points set couleur = 'blue'
where couleur = 'jaune';
|-
| || ||
begin;
update points set couleur = 'rouge'
where couleur = 'blue';
À ce point, au moins une des trois transactions est condamnée. Pour garantir que les traitement progressent, on attend qu'une valide. Le commit va réussir, ce qui non seulement garantit que les traitements progressent, mais qu'une tentative de reprendre une transaction échouée n'échouera pas ''sur la même combinaison de transactions''.
|-
|
commit;
Le premier commit gagne. La session 2 doit échouer à ce point, parce que durant le commit il a été déterminé qu'elle a les plus grandes chances de réussir si réessayée immédiatement.
select couleur, count(*) from points
group by couleur
order by couleur;

couleur | count
----------+-------
blue | 3000
jaune | 6000
(2 rows)
Cela semble avoir été exécuté avant les autres mises à jour.
|-
| || ||
commit;
Cela fonctionne si on l'essaye à ce moment. Si la session 2 effectue davantage de travail avant, cette transaction pourrait aussi devoir être annulée et réessayée.
select couleur, count(*) from points
group by couleur
order by couleur;

couleur | count
----------+-------
rouge | 3000
jaune | 6000
(2 rows)
Elle semble s'être exécutée après la transaction de la session 1.
|-
| ||
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
Une erreur de sérialisation. Nous annulons et réessayons.
rollback;
begin;
update points set couleur = 'blue'
where couleur = 'jaune';
commit;
Une nouvelle tentative réussira.
select couleur, count(*) from points
group by couleur
order by couleur;

couleur | count
---------+-------
blue | 6000
rouge | 3000
(2 rows)
Elle semble s'être exécutée en dernier, ce qu'elle a d'ailleurs fait.
|}
Un point intéressant est que si la session 2 avait tenté de valider après la session 1 et avant la session 3, elle aurait tout de même échoué, et une re-tentative aurait aussi réussi, mais le comportement de la transaction de la session 3 n'est pas déterministe. Elle pourrait avoir réussi, ou avoir reçu une erreur de sérialisation et avoir nécessité d'être rejouée.

C'est parce que le verrouillage de prédicat utilisé par le mécanisme de détection de conflit s'appuie sur les pages et enregistrement effectivement accédés, et il y a un facteur aléatoire utilisé lors de l'insertion des entrées d'index qui ont des clés égales, afin de réduire la contention; donc même avec des séquences d'évènements identiques il est toujours possible de voir des différences sur où les erreurs de sérialisation se produisent. C'est pour cela qu'il est important, quand on s'appuie sur les transactions sérialisables pour gérer la concurrence, d'avoir un système généralisé permettant d'identifier les erreurs de sérialisation et de rejouer les transactions depuis leur début.

Il convient aussi de noter que si la session 2 avait validé la seconde tentative de transaction avant que la session 3 ait validé sa transaction, toute requête ultérieure qui aurait vu des enregistrements mis à jour de jaune à bleu (et validés) aurait, de façon déterministe, fait échouer la transaction de la session 3, parce que ces enregistrements ne seraient pas des enregistrements que la session 3 verraient comme bleu et mettraient à jour à rouge. Pour que la transaction 3 réussisse, elle doit pouvoir être considérée comme ayant été exécutée avant la transaction validée de la session 2. Par conséquent, exposer un état dans lequel le travail de la transaction de la session 2 est visible, mais pas le travail de la transaction de la session 3 signifie que la transaction de la session 3 doit échouer. L'acte d' ''observer'' un état récemment modifié de la base peut entraîner des erreurs de sérialisation. Cela sera exploré plus avant dans d'autres exemples.

=== Mettre en place des règles métier dans des triggers ===

Si toutes les transactions sont sérialisables, des règles métier peuvent être vérifiées par des triggers sans les problèmes associés avec les autres niveaux d'isolation de transactions. Quand une contrainte déclarative fonctionne, elle sera en règle générale plus rapide, plus simple à implémenter et à maintenir, et moins sujette à bug - les triggers ne devront donc être utilisés comme suit que quand une contrainte déclarative ne fonctionnera pas.

----
==== Contraintes similaires à de l'unicité ====

Imaginons que vous vouliez quelque chose de similaire à une contrainte unique, mais en un peu plus compliqué. Pour cet exemple, nous voulons l'unicité des six premiers caractères de la colonne texte.

Cet exemple peut être mis en place avec les ordres suivants:
create table t (id int not null, val text not null);
with x (n) as (select generate_series(1,10000))
insert into t select x.n, md5(x.n::text) from x;
alter table t add primary key(id);
create index t_val on t (val);
vacuum analyze t;
create function t_func()
returns trigger
language plpgsql as $$
declare
st text;
begin
st := substring(new.val from 1 for 6);
if tg_op = 'UPDATE' and substring(old.val from 1 for 6) = st then
return new;
end if;
if exists (select * from t where val between st and st || 'z') then
raise exception 't.val pas unique sur les six premiers caractères: "%"', st;
end if;
return new;
end;
$$;
create trigger t_trig
before insert or update on t
for each row execute procedure t_func();

Pour vérifier que le trigger fait bien respecter la règle métier quand il n'y a pas de problème de concurrence, sur une connexion unique:

insert into t values (-1, 'this old dog');
insert into t values (-2, 'this old cat');

ERROR: t.val pas unique sur les six premiers caractères: "this o"

Essayons maintenant avec deux sessions concurrentes.

{|
|+ Exemple de contrainte similaire à de l'unicité
! session 1
! session 2
|-
|
begin;
insert into t values (-3, 'the river flows');
|-
| ||
begin;
insert into t values (-4, 'the right stuff');
Cela fonctionne pour le moment, parce que le travail de l'autre transaction n'est pas visible de cette transaction, mais les deux transactions ne peuvent pas valider sans violer la règle métier.
commit;
Le premier à valider gagne. La transaction est garantie.
|-
|
Un commit ici échouerait, ainsi que n'importe quel autre ordre qu'on tenterait d'exécuter dans cette transaction condamnée.
select * from t where id < 0;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Canceled on identification as a pivot,
during conflict out checking.
HINT: The transaction might succeed if retried.

Comme il s'agit d'une erreur de sérialisation, la transaction devrait être réessayée.

rollback;
begin;
insert into t values (-3, 'the river flows');

Lors de la nouvelle tentative, nous recevons une erreur plus utile à l'utilisateur.

ERROR: t.val pas unique sur les six premiers caractères: "the ri"
|}

----
==== Contraintes similaires à des clés étrangères ====

Quelquefois deux tables doivent avoir un lien très similaire à une relation de clé étrangère, mais il y a des critères supplémentaires qui rendrait la clé étrangère insuffisante à traiter la vérification d'intégrité nécessaire. Dans cet exemple un table project contient une référence à la clé d'une table person dans sa propre colonne project_manager, mais une personne ''quelconque'' ne suffira pas; la personne spécifiée doit être un gestionnaire de projet.

On peut mettre en place cet exemple avec les ordres suivants:
create table person
(
person_id int not null primary key,
person_name text not null,
is_project_manager boolean not null
);
create table project
(
project_id int not null primary key,
project_name text not null,
project_manager int not null
);
create index project_manager
on project (project_manager);

create function person_func()
returns trigger
language plpgsql as $$
begin
if tg_op = 'DELETE' and old.is_project_manager then
if exists (select * from project
where project_manager = old.person_id) then
raise exception
'une personne ne peut être supprimée tant qu''elle est responsable d''un projet';
end if;
end if;
if tg_op = 'UPDATE' then
if new.person_id is distinct from old.person_id then
raise exception 'il est interdit de modifier person_id';
end if;
if old.is_project_manager and not new.is_project_manager then
if exists (select * from project
where project_manager = old.person_id) then
raise exception
'une personne doit rester gestionnaire de projet tant qu''elle est responsable d''un projet';
end if;
end if;
end if;
if tg_op = 'DELETE' then
return old;
else
return new;
end if;
end;
$$;
create trigger person_trig
before update or delete on person
for each row execute procedure person_func();

create function project_func()
returns trigger
language plpgsql as $$
begin
if tg_op = 'INSERT'
or (tg_op = 'UPDATE' and new.project_manager <> old.project_manager) then
if not exists (select * from person
where person_id = new.project_manager
and is_project_manager) then
raise exception
'project_manager doit être défini en tant que gestionnaire de projet dans la table person';
end if;
end if;
return new;
end;
$$;
create trigger project_trig
before insert or update on project
for each row execute procedure project_func();

insert into person values (1, 'Kevin Grittner', true);
insert into person values (2, 'Peter Parker', true);
insert into project values (101, 'parallel processing', 1);
{|
|+ Exemple de contrainte similaire à une contrainte de clé étrangère
! session 1
! session 2
|-
|
Une personne est mise à jour pour ne plus être un gestionnaire de projet.
begin;
update person
set is_project_manager = false
where person_id = 2;
|-
| ||
En même temps, un projet est mis à jour afin que de rendre cette personne responsable de ce projet.
begin;
update project
set project_manager = 2
where project_id = 101;
Il n'est pas possible de valider les deux. Le premier à valider gagne.
commit;
L'affectation de la personne au projet valide d'abord, ce qui entraîne que l'autre transaction doit maintenant échouer. Si l'autre transaction s'était exécuté à un autre niveau d'isolation, les deux transactions auraient validé, entraînant une violation des règles métier.
|-
|
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
A serialization failure. We roll back and try again.
rollback;
begin;
update person
set is_project_manager = false
where person_id = 2;

ERROR: une personne doit rester gestionnaire de
projet tant qu'elle est responsable d'un projet
Lors de la seconde tentative, nous récupérons un message intelligible.
|}

=== Transactions en Lecture Seule ===

Bien qu'une transaction en lecture seule ne puisse contribuer à une anomalie qui persiste dans la base, dans le mode Repeatable Read implémenté par le SSI, elle peut "voir" un état qui n'est pas cohérent avec l'exécution sérialisée (une à la fois) des transactions. Une transaction Serializable implémentée avec SSI ne verra jamais ces anomalies transitoires.

----

==== Rapport de Dépôt ====

Une classe générale de problèmes invoquant des transactions en lecture seule est le traitement par lots, où une table contrôle quel lot (batch) est actuellement la cible des insertions. Un lot est fermé en mettant à jour la table de contrôle, point à partir duquel le lot est considéré comme "verrouillé" contre tout changement ultérieur, et le traitement de ce lot se produit.

Ce genre de problématique peut être trouvé de façon concrète dans le traitement de reçus. Des reçus peuvent être ajoutés à un lot identifié par une date de dépôt, ou (si plus d'un dépôt par jour est possible) un numéro de lot de reçu abstrait. Un un point durant la journée, alors que la banque est toujours ouverte, le lot est fermé, un rapport de l'argent reçu est imprimé, et l'argent est emmené à la banque pour y être déposé.

L'exemple peut être mis en place avec ces ordres:
create table control
(
deposit_no int not null
);
insert into control values (1);
create table receipt
(
receipt_no serial primary key,
deposit_no int not null,
payee text not null,
amount money not null
);
insert into receipt
(deposit_no, payee, amount)
values ((select deposit_no from control), 'Crosby', '100');
insert into receipt
(deposit_no, payee, amount)
values ((select deposit_no from control), 'Stills', '200');
insert into receipt
(deposit_no, payee, amount)
values ((select deposit_no from control), 'Nash', '300');
{|
|+ Exemple de Rapport de Dépôt
! session 1
! session 2
|-
|
Au comptoir de réception, un autre reçu est ajouté au lot courant.
begin; -- T1
insert into receipt
(deposit_no, payee, amount)
values
(
(select deposit_no from control),
'Young', '100'
);
Cette transaction peut voir son propre insert, mais il n'est pas visible pour les autres transactions jusqu'à sa validation.
select * from receipt;

receipt_no | deposit_no | payee | amount
------------+------------+--------+---------
1 | 1 | Crosby | $100.00
2 | 1 | Stills | $200.00
3 | 1 | Nash | $300.00
4 | 1 | Young | $100.00
(4 rows)
|-
| ||
À peu près au même moment, un superviseur clique sur un bouton pour fermer le lot de reçus.
begin; -- T2
select deposit_no from control;

deposit_no
------------
1
(1 row)
L'application note le lot de reçus qui est sur le point d'être fermé, incrémente le numéro de lot, et l'enregistre dans la table de contrôle.
update control set deposit_no = 2;
commit;
T1, la transaction qui insère le dernier reçu du dernier lot, n'a pas encore validé, bien que le lot ait été fermé. Si T1 valide avant que quelqu'un ne regarde le contenu du lot, tout va bien. Pour le moment nous n'avons aucun problème; le reçu "a l'air" d'avoir été ajouté avant que le lot ait été fermé. Nous avons un comportement qui est cohérent avec une exécution "une par une" des transactions: T1 -> T2.

Pour le besoin de la démonstration, nous allons déclencher le rapport de dépôt avant que le dernier reçu ne soit validé.
begin; -- T3
select * from receipt where deposit_no = 1;

receipt_no | deposit_no | payee | amount
------------+------------+--------+---------
1 | 1 | Crosby | $100.00
2 | 1 | Stills | $200.00
3 | 1 | Nash | $300.00
(3 rows)
Maintenant nous avons un problème. T3 a été démarré en sachant que T2 a été validée, donc T3 doit être considérée comme ayant exécutée avant T2. (cela aurait pu aussi être vrai si T3 avait été lancé indépendamment et avait lu la table de contrôle, voyant le nouveau deposit_no.) Mais T3 ne peut pas voir le travail de T1, donc T1 a l'air d'avoir été exécuté après T3. Nous avons donc une boucle T1 -> T2 -> T3 -> T1. Et cela poserait problème en termes pratiques; le lot est censé être fermé et immuable, mais une modification apparaîtra sur le tard -- peut être après le voyage à la banque.

Au niveau d'isolation REPEATABLE READ cela se déroulerait sans message d'erreur, sans que l'anomalie ne soit détectée. Au niveau d'isolation SERIALIZABLE une des transactions serait annulée pour préserver l'intégrité du système. Puisqu'une annulation de T3 entraînerait à nouveau la même erreur si T1 était encore active, PostgreSQL va annuler T1, pour qu'une nouvelle tentative ayant lieu immédiatement puisse réussir.
|-
|
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
OK, let's retry.
rollback;
begin; -- T1 retry
insert into receipt
(deposit_no, payee, amount)
values
(
(select deposit_no from control),
'Young', '100'
);

À quoi ressemble la table reçu maintenant?

select * from receipt;

receipt_no | deposit_no | payee | amount
------------+------------+--------+---------
1 | 1 | Crosby | $100.00
2 | 1 | Stills | $200.00
3 | 1 | Nash | $300.00
5 | 2 | Young | $100.00
(4 rows)

Le reçu est maintenant dans le nouveau lot, rendant le rapport de dépôt de T3 correct!

commit;

Plus de problème maintenant.
|-
| ||
commit;
Cela n'aurait posé aucun problème à n'importe quel moment après le SELECT de T3.
|}

[[Category:Français]]

SSI

Kgrittn — Sun, 27 Nov 2011 00:06:42 GMT

Kgrittn: /* Deposit Report */ Fix typo.

{{Languages}}

Documentation of Serializable Snapshot Isolation (SSI) in PostgreSQL compared to plain Snapshot Isolation (SI). These correspond to the SERIALIZABLE and REPEATABLE READ transaction isolation levels, respectively, in PostgreSQL beginning with version 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document shows problems which can occur with certain combinations of transactions at the REPEATABLE READ transaction isolation level, and how they are avoided at the SERIALIZABLE transaction isolation level beginning with PostgreSQL version 9.1.

This document is oriented toward the application programmer or database administrator. For internals of the SSI implementation, please see the [[Serializable]] Wiki page. For more information about how to use this isolation level, see [http://www.postgresql.org/docs/current/interactive/transaction-iso.html#XACT-SERIALIZABLE the current PostgreSQL documentation].

== Examples ==

In environments which avoid blocking-based integrity protection by counting on SSI, it will be common for the database to be configured (in postgresql.conf) with:
default_transaction_isolation = 'serializable'
Because of this, all examples were tested with this setting, and clutter is avoided by using a simple "begin" without explicitly declaring the transaction isolation level for each transaction.

=== Simple Write Skew ===

When two concurrent transactions each determine what they are writing based on reading a data set which overlaps what the other is writing, you can get a state which could not occur if either had run before the other. This is known as ''write skew'', and is the simplest form of serialization anomaly against which SSI protects you.

When there is write skew in SSI, both transactions proceed until one transaction commits. The first committer wins and the other transaction is rolled back. The "first committer wins" rule ensures that there is progress and that the transaction which is rolled back can immediately be retried.

----
==== Black and White ====

In this case there are rows with a color column containing 'black' or 'white'. Two users concurrently try to make all rows contain matching color values, but their attempts go in opposite directions. One is trying to update all white rows to black and the other is trying to update all black rows to white.

If these updates are run serially, all colors will match. If they are run concurrently in REPEATABLE READ mode, the values will be switched, which is not consistent with any serial order of runs. If they are run concurrently in SERIALIZABLE mode, SSI will notice the write skew and roll back one of the transactions.

The example can be set up with these statements:
create table dots
(
id int not null primary key,
color text not null
);
insert into dots
with x(id) as (select generate_series(1,10))
select id, case when id % 2 = 1 then 'black'
else 'white' end from x;
{|
|+ Black and White Example
! session 1
! session 2
|-
|
begin;
update dots set color = 'black'
where color = 'white';
|-
| ||
begin;
update dots set color = 'white'
where color = 'black';
At this point one transaction or the other is doomed to fail.
commit;
First commit wins.
select * from dots order by id;

id | color
----+-------
1 | white
2 | white
3 | white
4 | white
5 | white
6 | white
7 | white
8 | white
9 | white
10 | white
(10 rows)
This one ran as if by itself.
|-
|
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
A serialization failure. We roll back and try again.
rollback;
begin;
update dots set color = 'black'
where color = 'white';
commit;
No concurrent transaction to interfere.
select * from dots order by id;

id | color
----+-------
1 | black
2 | black
3 | black
4 | black
5 | black
6 | black
7 | black
8 | black
9 | black
10 | black
(10 rows)
This transaction ran by itself, after the other.
|}

----
==== Intersecting Data ====

This example is taken from the PostgreSQL documentation. Two concurrent transactions read data, and each uses it to update the range read by the other. A simple, though somewhat contrived, example of data skew.

The example can be set up with these statements:
CREATE TABLE mytab
(
class int NOT NULL,
value int NOT NULL
);
INSERT INTO mytab VALUES
(1, 10), (1, 20), (2, 100), (2, 200);
{|
|+ Intersecting Data Example
! session 1
! session 2
|-
|
BEGIN;
SELECT SUM(value) FROM mytab WHERE class = 1;

sum
-----
30
(1 row)

INSERT INTO mytab VALUES (2, 30);
|-
| ||
BEGIN;
SELECT SUM(value) FROM mytab WHERE class = 2;

sum
-----
300
(1 row)

INSERT INTO mytab VALUES (1, 300);
Each transaction has modified what the other transaction would have read. If both were allowed to commit, this would break serializable behavior, because if they were run one at a time, one of the transactions would have seen the INSERT the other committed. We wait for a successful COMMIT of one of the transactions before we roll anything back, though, to ensure progress and prevent thrashing.
COMMIT;
|-
|
COMMIT;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
So now we roll back the failed transaction and retry it from the beginning.
ROLLBACK;
BEGIN;
SELECT SUM(value) FROM mytab WHERE class = 1;

sum
-----
330
(1 row)

INSERT INTO mytab VALUES (2, 330);
COMMIT;
This succeeds, leaving an end result consistent with a serial execution of the transactions.
SELECT * FROM mytab;

class | value
-------+-------
1 | 10
1 | 20
2 | 100
2 | 200
1 | 300
2 | 330
(6 rows)
|}

----
==== Overdraft Protection ====

The hypothetical case is that there is a bank which allows depositors to withdraw money up to the total of what they have in all accounts. The bank will later automatically transfer funds as needed to close the day with a positive balance in each account. Within a single transaction they check that the total of all accounts exceeds the amount requested.

Someone's trying to get clever and trick the bank by submitting $900 withdrawals to two accounts with $500 balances simultaneously. At the REPEATABLE READ transaction isolation level, that could work; but if the SERIALIZABLE transaction isolation level is used, SSI will detect a "dangerous structure" in the read/write pattern and reject one of the transactions.

The example can be set up with these statements:

create table account
(
name text not null,
type text not null,
balance money not null default '0.00'::money,
primary key (name, type)
);
insert into account values
('kevin','saving', 500),
('kevin','checking', 500);

{|
|+ Overdraft Protection Example
! session 1
! session 2
|-
|
begin;
select type, balance from account
where name = 'kevin';

type | balance
----------+---------
saving | $500.00
checking | $500.00
(2 rows)
The total is $1000, so a $900 withdrawal is OK.
|-
| ||
begin;
select type, balance from account
where name = 'kevin';

type | balance
----------+---------
saving | $500.00
checking | $500.00
(2 rows)
The total is $1000, so a $900 withdrawal is OK.
|-
|
update account
set balance = balance - 900::money
where name = 'kevin' and type = 'saving';
So far everything's OK.
|-
| ||
update account
set balance = balance - 900::money
where name = 'kevin' and type = 'checking';
Now we have a problem. This can't co-exist with the other transaction's activity. We don't cancel yet, because the transaction would fail on the same conflicts if retried. The first committer will win, and the other transaction will fail when it tries to continue after that.
|-
|
commit;
This one happened to commit first. Its work is persisted.
|-
| ||
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
This transaction failed to withdraw the money.
Now we roll back and retry the transaction.

rollback;
begin;
select type, balance from account
where name = 'kevin';

type | balance
----------+----------
saving | -$400.00
checking | $500.00
(2 rows)
We see they have a net of $100. This request for $900 will be rejected by the application.
|}

=== Three or More Transactions ===

Serialization anomalies can result from more complex patterns of access, involving three or more transactions.

----
==== Primary Colors ====

This is similar to the "Black and White" write skew example, except that we're using the three primary colors. One transaction is trying to update red to yellow, the next is trying to update yellow to blue, and the third is trying to update blue to red. If these were executed one at a time, you would be left with either one or two colors in the table, depending on the order of execution. If any two are executed concurrently, the one trying to read the rows being updated by the other will appear to execute first, since it won't see the work of the other transaction, so there is no problem there. Whether the other transaction is run before that or after that, the results are consistent with some serial order of execution.

If all three are run concurrently, there is a cycle in the apparent order of execution. A Repeatable Read transaction would not detect this, and the table would still have three colors. A Serializable transaction will detect the problem and roll one of the transactions back with a serialization failure.

The example can be set up with these statements:
create table dots
(
id int not null primary key,
color text not null
);
insert into dots
with x(id) as (select generate_series(1,9000))
select id, case when id % 3 = 1 then 'red'
when id % 3 = 2 then 'yellow'
else 'blue' end from x;
create index dots_color on dots (color);
analyze dots;
{|
|+ Primary Colors Example
! session 1
! session 2
! session 3
|-
|
begin;
update dots set color = 'yellow'
where color = 'red';
|-
| ||
begin;
update dots set color = 'blue'
where color = 'yellow';
|-
| || ||
begin;
update dots set color = 'red'
where color = 'blue';
At this point at least one of these three transactions is doomed to fail. To ensure progress, we wait until one of them commits. The commit will succeed, which will not only ensure that progress is made but that an immediate retry of a failed transaction won't fail again ''on the same combination of transactions''.
|-
|
commit;
First commit wins. Session 2 is bound to fail at this point, because during commit it was determined that it had the better chance of succeeding if retried immediately.
select color, count(*) from dots
group by color
order by color;

color | count
--------+-------
blue | 3000
yellow | 6000
(2 rows)
This appears to have run before the other updates.
|-
| || ||
commit;
This works if attempted at this point. If session 2 does more work first, this transaction might also need to be cancelled and retried.
select color, count(*) from dots
group by color
order by color;

color | count
--------+-------
red | 3000
yellow | 6000
(2 rows)
This appears to have run after the transaction on session 1.
|-
| ||
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
A serialization failure. We roll back and try again.
rollback;
begin;
update dots set color = 'blue'
where color = 'yellow';
commit;
Things are OK on retry.
select color, count(*) from dots
group by color
order by color;

color | count
-------+-------
blue | 6000
red | 3000
(2 rows)
This appears to have run last, which it did.
|}
An interesting point is that if session 2 attempted to commit after session 1 and before session 3, it would still have failed, and a retry would still have succeeded, but the fate of the transaction on session 3 is not deterministic. It might have succeeded or it might have gotten a serialization failure and required a retry.

This is because the predicate locking used as part of conflict detection works based on pages and tuples actually accessed, and there is a random factor used in inserting index entries which have equal keys, in order to minimize contention; so even with identical run sequences it is possible to see differences in where serialization failures occur. That is why it is important, when relying on serializable transactions for managing concurrency, to have some generalized technique for identifying serialization failures and retrying transactions from the beginning.

It is also worth noting that if session 2 committed the retry transaction before session 3 committed its transaction, any subsequent query which viewed rows successfully updated from yellow to blue by session 2 would deterministically doom the transaction on session 3, because these would not be rows which session 3 would see as blue and update to red. For the transaction on session 3 to successfully commit, it must be considered to have run before the successful transaction on session 2, so exposing a state in which the work of the transaction on session 2 is visible, but not the work of the transaction on session 3, means that the transaction on session 3 must fail. The act of ''observing'' a recently modified database state can cause serialization failures. This will be further explored in other examples.

=== Enforcing Business Rules in Triggers ===

If all transactions are serializable, business rules can be enforced in triggers without the problems associated with other transaction isolation levels. Where a declarative constraint works, it will generally be faster, easier to implement and maintain, and less prone to bugs -- so triggers should only be used this way where a declarative constraint won't work.

----
==== Unique-Like Constraints ====

Say you want something similar to a unique constraint, but it's a little more complicated. For this example, we want uniqueness in the first six characters of the text column.

The example can be set up with these statements:
create table t (id int not null, val text not null);
with x (n) as (select generate_series(1,10000))
insert into t select x.n, md5(x.n::text) from x;
alter table t add primary key(id);
create index t_val on t (val);
vacuum analyze t;
create function t_func()
returns trigger
language plpgsql as $$
declare
st text;
begin
st := substring(new.val from 1 for 6);
if tg_op = 'UPDATE' and substring(old.val from 1 for 6) = st then
return new;
end if;
if exists (select * from t where val between st and st || 'z') then
raise exception 't.val not unique on first six characters: "%"', st;
end if;
return new;
end;
$$;
create trigger t_trig
before insert or update on t
for each row execute procedure t_func();

To confirm that the trigger is enforcing the business rule when there are no concurrency issues, on a single connection:

insert into t values (-1, 'this old dog');
insert into t values (-2, 'this old cat');

ERROR: t.val not unique on first six characters: "this o"

Now we try in two concurrent sessions.

{|
|+ Unique-Like Constraint Example
! session 1
! session 2
|-
|
begin;
insert into t values (-3, 'the river flows');
|-
| ||
begin;
insert into t values (-4, 'the right stuff');
This works for the moment, because the work of the other transaction is not visible to this transaction, but both transactions may not commit without violating the business rule.
commit;
The first committer wins. This transaction is safe.
|-
|
A commit here would fail, but so would an attempt to run any other statement within this doomed transaction.
select * from t where id < 0;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Canceled on identification as a pivot,
during conflict out checking.
HINT: The transaction might succeed if retried.

Since this is a serialization failure, the transaction should be retried.

rollback;
begin;
insert into t values (-3, 'the river flows');

On retry we get an error which is more helpful to the user.

ERROR: t.val not unique on first six characters: "the ri"
|}

----
==== FK-Like Constraints ====

Sometimes two tables must have a relationship very similar to a foreign key relationship, but there are extra criteria which makes a foreign key insufficient to completely cover the necessary integrity checking. In this example a project table contains a reference to a person table's key in a project_manager column, but not just ''any'' person will do; the person specified must be flagged as a project manager.

The example can be set up with these statements:
create table person
(
person_id int not null primary key,
person_name text not null,
is_project_manager boolean not null
);
create table project
(
project_id int not null primary key,
project_name text not null,
project_manager int not null
);
create index project_manager
on project (project_manager);

create function person_func()
returns trigger
language plpgsql as $$
begin
if tg_op = 'DELETE' and old.is_project_manager then
if exists (select * from project
where project_manager = old.person_id) then
raise exception
'person cannot be deleted while manager of any project';
end if;
end if;
if tg_op = 'UPDATE' then
if new.person_id is distinct from old.person_id then
raise exception 'change to person_id is not allowed';
end if;
if old.is_project_manager and not new.is_project_manager then
if exists (select * from project
where project_manager = old.person_id) then
raise exception
'person must remain a project manager while managing any projects';
end if;
end if;
end if;
if tg_op = 'DELETE' then
return old;
else
return new;
end if;
end;
$$;
create trigger person_trig
before update or delete on person
for each row execute procedure person_func();

create function project_func()
returns trigger
language plpgsql as $$
begin
if tg_op = 'INSERT'
or (tg_op = 'UPDATE' and new.project_manager <> old.project_manager) then
if not exists (select * from person
where person_id = new.project_manager
and is_project_manager) then
raise exception
'project_manager must be defined as a project manager in the person table';
end if;
end if;
return new;
end;
$$;
create trigger project_trig
before insert or update on project
for each row execute procedure project_func();

insert into person values (1, 'Kevin Grittner', true);
insert into person values (2, 'Peter Parker', true);
insert into project values (101, 'parallel processing', 1);
{|
|+ FK-Like Constraints Example
! session 1
! session 2
|-
|
One person is being updated to no longer be a project manager.
begin;
update person
set is_project_manager = false
where person_id = 2;
|-
| ||
At the same time, a project is being updated to make that person the manager of that project.
begin;
update project
set project_manager = 2
where project_id = 101;
These can't both be committed. The first commit will win.
commit;
The assignment of the person to the project commits first, so the other transaction is now doomed to fail. If either transaction had run at a different isolation level, both transactions could have committed, resulting in a violation of the business rules.
|-
|
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
A serialization failure. We roll back and try again.
rollback;
begin;
update person
set is_project_manager = false
where person_id = 2;

ERROR: person must remain a project manager
while managing any projects
On the retry we get a meaningful message.
|}

=== Read Only Transactions ===

While a Read Only transaction cannot contribute to an anomaly which persists in the database, under Repeatable Read transaction isolation it can ''see'' a state which is not consistent with any serial (one-at-a-time) execution of transactions. A Serializable transaction implemented with SSI will never see such transient anomalies.

----

==== Deposit Report ====

A general class of problems involving read only transactions is batch processing, where one table controls which batch is currently the target of inserts. A batch is closed by updating the control table, at which point the batch is considered "locked" against further change, and processing of that batch occurs.

A particular example of this which occurs in real-world bookkeeping is receipting. Receipts might be added to a batch identified by the deposit date, or (if more than one deposit per day is possible) an abstract receipt batch number. At some point during the day, while the bank is still open, the batch is closed, a report is printed of the money received, and the money is taken to the bank for deposit.

The example can be set up with these statements:
create table control
(
deposit_no int not null
);
insert into control values (1);
create table receipt
(
receipt_no serial primary key,
deposit_no int not null,
payee text not null,
amount money not null
);
insert into receipt
(deposit_no, payee, amount)
values ((select deposit_no from control), 'Crosby', '100');
insert into receipt
(deposit_no, payee, amount)
values ((select deposit_no from control), 'Stills', '200');
insert into receipt
(deposit_no, payee, amount)
values ((select deposit_no from control), 'Nash', '300');
{|
|+ Deposit Report Example
! session 1
! session 2
|-
|
At a receipting counter, another receipt is added to the current batch.
begin; -- T1
insert into receipt
(deposit_no, payee, amount)
values
(
(select deposit_no from control),
'Young', '100'
);
This transaction can see its own insert, but it's not visible to other transactions until commit.
select * from receipt;

receipt_no | deposit_no | payee | amount
------------+------------+--------+---------
1 | 1 | Crosby | $100.00
2 | 1 | Stills | $200.00
3 | 1 | Nash | $300.00
4 | 1 | Young | $100.00
(4 rows)
|-
| ||
At about the same time, a supervisor clicks a button to close the receipt batch.
begin; -- T2
select deposit_no from control;

deposit_no
------------
1
(1 row)
The application notes the receipting batch which is about to be closed, increments the batch number, and saves that to the control table.
update control set deposit_no = 2;
commit;
T1, the transaction inserting the last receipt for the old batch, hasn't committed yet even though the batch has been closed. If T1 commits before anyone looks at the contents of the closed batch, everything is OK. So far we don't have a problem; the receipt ''appears'' to have added before the batch was closed. We have behavior which is consistent with some one-at-a-time execution of the transactions: T1 -> T2.

For purposes of demonstration, we'll have the deposit report start before that last receipt commits.
begin; -- T3
select * from receipt where deposit_no = 1;

receipt_no | deposit_no | payee | amount
------------+------------+--------+---------
1 | 1 | Crosby | $100.00
2 | 1 | Stills | $200.00
3 | 1 | Nash | $300.00
(3 rows)
Now we have a problem. T3 was started with the knowledge that T2 committed successfully, so T3 must be considered to have run after T2. (This could also be true if the T3 had run independently and selected from the control table, seeing the updated deposit_no.) But T3 cannot see the work of T1, so T1 appears to have run after T3. So we have a cycle T1 -> T2 -> T3 -> T1. And this would be a problem in practical terms; the batch is supposed to be closed and immutable, but a change will pop up late -- perhaps after the trip to the bank.

Under the REPEATABLE READ isolation level this would silently proceed without the anomaly being noticed. Under the SERIALIZABLE isolation level one of the transactions will be rolled back to protect the integrity of the system. Since a rollback and retry of T3 would hit the same error if T1 was still active, PostgreSQL will cancel T1, so that an immediate retry will succeed.
|-
|
commit;

ERROR: could not serialize access
due to read/write dependencies
among transactions
DETAIL: Cancelled on identification
as a pivot, during commit attempt.
HINT: The transaction might succeed if retried.
OK, let's retry.
rollback;
begin; -- T1 retry
insert into receipt
(deposit_no, payee, amount)
values
(
(select deposit_no from control),
'Young', '100'
);

What does the receipt table look like now?

select * from receipt;

receipt_no | deposit_no | payee | amount
------------+------------+--------+---------
1 | 1 | Crosby | $100.00
2 | 1 | Stills | $200.00
3 | 1 | Nash | $300.00
5 | 2 | Young | $100.00
(4 rows)

The receipt now falls into the next batch, making the deposit report in T3 correct!

commit;

No problem now.
|-
| ||
commit;
This would have been OK anytime after T3's SELECT.
|}

Serializable

Kgrittn — Fri, 25 Nov 2011 23:22:00 GMT

Kgrittn: /* SSI Algorithm */ Fix references to T1 in proof; they should have been Tin. Subscript the zero in T0.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]]. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[[#SQL92|<nowiki>[4]</nowiki>]]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation[[#Cahill2009|<nowiki>[2]</nowiki>]] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [[#Cahill2009|<nowiki>[2]</nowiki>]]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because Tin was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before Tin started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before Tin starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.
* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.
* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.
* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.
* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.
* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.
* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".
* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.
* Modifying a heap tuple creates a rw-conflict with any transaction that holds a SIREAD lock on that tuple, or on the page or relation that contains it.
* Inserting a new tuple creates a rw-conflict with any transaction holding a SIREAD lock on the entire relation. It doesn't conflict with page-level locks, because page-level locks are only used to aggregate tuple locks. Unlike index page locks, they don't lock "gaps" on the page.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[[#CahillEtAl2008|<nowiki>[1]</nowiki>]], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper[[#Foundations2007|<nowiki>[3]</nowiki>]], along with the locking papers referenced from that and the Cahill papers[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]].

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.
* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.
* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.
* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.
* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.
* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 23:14:41 GMT

Kgrittn: /* SSI Algorithm */ Pretty up the ASCII-art diagram of a dangerous structure a little.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]]. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[[#SQL92|<nowiki>[4]</nowiki>]]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation[[#Cahill2009|<nowiki>[2]</nowiki>]] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [[#Cahill2009|<nowiki>[2]</nowiki>]]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.
* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.
* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.
* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.
* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.
* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.
* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".
* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.
* Modifying a heap tuple creates a rw-conflict with any transaction that holds a SIREAD lock on that tuple, or on the page or relation that contains it.
* Inserting a new tuple creates a rw-conflict with any transaction holding a SIREAD lock on the entire relation. It doesn't conflict with page-level locks, because page-level locks are only used to aggregate tuple locks. Unlike index page locks, they don't lock "gaps" on the page.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[[#CahillEtAl2008|<nowiki>[1]</nowiki>]], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper[[#Foundations2007|<nowiki>[3]</nowiki>]], along with the locking papers referenced from that and the Cahill papers[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]].

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.
* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.
* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.
* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.
* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.
* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 23:06:26 GMT

Kgrittn: /* Index AM implementations */ Remove empty lines between points to prevent a separate unordered list per point.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]]. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[[#SQL92|<nowiki>[4]</nowiki>]]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation[[#Cahill2009|<nowiki>[2]</nowiki>]] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [[#Cahill2009|<nowiki>[2]</nowiki>]]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.
* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.
* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.
* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.
* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.
* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.
* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".
* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.
* Modifying a heap tuple creates a rw-conflict with any transaction that holds a SIREAD lock on that tuple, or on the page or relation that contains it.
* Inserting a new tuple creates a rw-conflict with any transaction holding a SIREAD lock on the entire relation. It doesn't conflict with page-level locks, because page-level locks are only used to aggregate tuple locks. Unlike index page locks, they don't lock "gaps" on the page.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[[#CahillEtAl2008|<nowiki>[1]</nowiki>]], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper[[#Foundations2007|<nowiki>[3]</nowiki>]], along with the locking papers referenced from that and the Cahill papers[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]].

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.
* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.
* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.
* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.
* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.
* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 23:04:46 GMT

Kgrittn: /* PostgreSQL Implementation */ Eliminate empty lines so points are a single unordered list, rather than a separate list for each point.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]]. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[[#SQL92|<nowiki>[4]</nowiki>]]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation[[#Cahill2009|<nowiki>[2]</nowiki>]] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [[#Cahill2009|<nowiki>[2]</nowiki>]]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.
* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.
* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.
* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.
* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.
* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.
* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".
* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.
* Modifying a heap tuple creates a rw-conflict with any transaction that holds a SIREAD lock on that tuple, or on the page or relation that contains it.
* Inserting a new tuple creates a rw-conflict with any transaction holding a SIREAD lock on the entire relation. It doesn't conflict with page-level locks, because page-level locks are only used to aggregate tuple locks. Unlike index page locks, they don't lock "gaps" on the page.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[[#CahillEtAl2008|<nowiki>[1]</nowiki>]], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper[[#Foundations2007|<nowiki>[3]</nowiki>]], along with the locking papers referenced from that and the Cahill papers[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]].

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.

* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.

* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.

* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.

* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.

* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 22:57:10 GMT

Kgrittn: /* PostgreSQL Implementation */ Add two implementation points from Dan's README-SSI.patch.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]]. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[[#SQL92|<nowiki>[4]</nowiki>]]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation[[#Cahill2009|<nowiki>[2]</nowiki>]] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [[#Cahill2009|<nowiki>[2]</nowiki>]]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.

* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.

* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.

* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.

* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.

* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.

* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".

* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.

* Modifying a heap tuple creates a rw-conflict with any transaction that holds a SIREAD lock on that tuple, or on the page or relation that contains it.

* Inserting a new tuple creates a rw-conflict with any transaction holding a SIREAD lock on the entire relation. It doesn't conflict with page-level locks, because page-level locks are only used to aggregate tuple locks. Unlike index page locks, they don't lock "gaps" on the page.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[[#CahillEtAl2008|<nowiki>[1]</nowiki>]], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper[[#Foundations2007|<nowiki>[3]</nowiki>]], along with the locking papers referenced from that and the Cahill papers[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]].

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.

* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.

* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.

* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.

* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.

* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 22:49:42 GMT

Kgrittn: /* Publications */ Terminate unclosed .

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]]. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[[#SQL92|<nowiki>[4]</nowiki>]]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation[[#Cahill2009|<nowiki>[2]</nowiki>]] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [[#Cahill2009|<nowiki>[2]</nowiki>]]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.

* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.

* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.

* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.

* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.

* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.

* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".

* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[[#CahillEtAl2008|<nowiki>[1]</nowiki>]], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper[[#Foundations2007|<nowiki>[3]</nowiki>]], along with the locking papers referenced from that and the Cahill papers[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]].

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.

* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.

* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.

* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.

* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.

* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 22:47:31 GMT

Kgrittn: Create anchors for footnotes and turn references into links (superscripted where appropriate).

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]]. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[[#SQL92|<nowiki>[4]</nowiki>]]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation[[#Cahill2009|<nowiki>[2]</nowiki>]] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [[#Cahill2009|<nowiki>[2]</nowiki>]]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.

* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.

* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.

* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.

* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.

* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.

* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".

* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[[#CahillEtAl2008|<nowiki>[1]</nowiki>]], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper[[#Foundations2007|<nowiki>[3]</nowiki>]], along with the locking papers referenced from that and the Cahill papers[[#CahillEtAl2008|<nowiki>[1]</nowiki>]][[#Cahill2009|<nowiki>[2]</nowiki>]].

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.

* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.

* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.

* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.

* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.

* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 21:56:26 GMT

Kgrittn: /* PostgreSQL Implementation */ Minor cleanup.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008<nowiki>[1][2]</nowiki>. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[4]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation [2] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [2]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

Notable aspects of the PostgreSQL implementation of SSI include:

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.

* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.

* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.

* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level retains its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.

* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.

* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.

* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".

* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[1], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper<nowiki>[3]</nowiki>, along with the locking papers referenced from that and the Cahill papers<nowiki>[1][2]</nowiki>.

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.

* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.

* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.

* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.

* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.

* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 21:46:47 GMT

Kgrittn: /* SSI Algorithm */ Minor spacing adjustment to new code.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008<nowiki>[1][2]</nowiki>. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[4]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation [2] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

::Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [2]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

The implementation of serializable transactions for PostgreSQL is accomplished through Serializable Snapshot Isolation (SSI), based on the work of Cahill, et al[1][2]. Fundamentally, this allows snapshot isolation to run as it has, with monitoring for conditions which could create a serialization anomaly.

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.

* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.

* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.

* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level will retain its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.

* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.

* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.

* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".

* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[1], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper<nowiki>[3]</nowiki>, along with the locking papers referenced from that and the Cahill papers<nowiki>[1][2]</nowiki>.

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.

* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.

* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.

* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.

* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.

* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.

Serializable

Kgrittn — Fri, 25 Nov 2011 21:38:31 GMT

Kgrittn: /* SSI Algorithm */ Fill in from Dan's README-SSI patch, with some Wiki formatting.

Information about the SSI implementation for the SERIALIZABLE transaction isolation level in PostgreSQL, new in release 9.1.

== Overview ==

With true serializable transactions, if you can show that your transaction will do the right thing if there are no concurrent transactions, it will do the right thing in any mix of serializable transactions or be rolled back with a serialization failure.

This document is oriented toward the techniques used to implement the feature in PostgreSQL. For information oriented toward application programmers and database administrators, see the [[SSI]] Wiki page.

=== Serializable and Snapshot Transaction Isolation Levels ===

Serializable transaction isolation is attractive for shops with active development by many programmers against a complex schema because it guarantees data integrity with very little staff time -- if a transaction can be shown to always do the right thing when it is run alone (before or after any other transaction), it will always do the right thing in any mix of concurrent serializable transactions. Where conflicts with other transactions would result in an inconsistent state within the database or an inconsistent view of the data, a serializable transaction will block or roll back to prevent the anomaly. The SQL standard provides a specific SQLSTATE for errors generated when a transaction rolls back for this reason, so that transactions can be retried automatically.

Before version 9.1, PostgreSQL did not support a full serializable isolation level. A request for serializable transaction isolation actually provided snapshot isolation. This has well known anomalies which can allow data corruption or inconsistent views of the data during concurrent transactions; although these anomalies only occur when certain patterns of read-write dependencies exist within a set of concurrent transactions. Where these patterns exist, the anomalies can be prevented by introducing conflicts through explicitly programmed locks or otherwise unnecessary writes to the database. Snapshot isolation is popular because performance is better than serializable isolation and the integrity guarantees which it does provide allow anomalies to be avoided or managed with reasonable effort in many environments.

[[Image:Serialization-Anomalies-in-Snapshot-Isolation.png|600px|center]]

=== Serializable Isolation Implementation Strategies ===

Techniques for implementing full serializable isolation have been published and in use in many database products for decades. The primary technique which has been used is Strict Two-Phase Locking (S2PL), which operates by blocking writes against data which has been read by concurrent transactions and blocking any access (read or write) against data which has been written by concurrent transactions. A cycle in a graph of blocking indicates a deadlock, requiring a rollback. Blocking and deadlocks under S2PL in high contention workloads can be debilitating, crippling throughput and response time.

A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008<nowiki>[1][2]</nowiki>. This technique, known as Serializable Snapshot Isolation (SSI) has many of the advantages of snapshot isolation. In particular, reads don't block anything and writes don't block reads. Essentially, it runs snapshot isolation but monitors the read-write conflicts between transactions to identify dangerous structures in the transaction graph which indicate that a set of concurrent transactions might produce an anomaly, and rolls back transactions to ensure that no anomalies occur. It will produce some false positives (where a transaction is rolled back even though there would not have been an anomaly), but will never let an anomaly occur. In the two known prototype implementations, performance for many workloads (even with the need to restart transactions which are rolled back) is very close to snapshot isolation and generally far better than an S2PL implementation.

=== Apparent Serial Order of Execution ===

One way to understand when snapshot anomalies can occur, and to visualize the difference between the serializable implementations described above, is to consider that among transactions executing at the serializable transaction isolation level, the results are required to be consistent with ''some'' serial (one-at-a-time) execution of the transactions[4]. How is that order determined in each?

In S2PL, each transaction locks any data it accesses. It holds the locks until committing, preventing other transactions from making conflicting accesses to the same data in the interim. Some transactions may have to be rolled back to prevent deadlock. But successful transactions can always be viewed as having occurred sequentially, in the order they committed.

With snapshot isolation, reads never block writes, nor vice versa, so more concurrency is possible. The order in which transactions appear to have executed is determined by something more subtle than in S2PL: read/write dependencies. If a transaction reads data, it appears to execute after the transaction that wrote the data it is reading. Similarly, if it updates data, it appears to execute after the transaction that wrote the previous version. These dependencies, which we call "wr-dependencies" and "ww-dependencies", are consistent with the commit order, because the first transaction must have committed before the second starts. However, there can also be dependencies between two *concurrent* transactions, i.e. where one was running when the other acquired its snapshot. These "rw-conflicts" occur when one transaction attempts to read data which is not visible to it because the transaction which wrote it (or will later write it) is concurrent. The reading transaction appears to have executed first, regardless of the actual sequence of transaction starts or commits, because it sees a database state prior to that in which the other transaction leaves it.

Anomalies occur when a cycle is created in the graph of dependencies: when a dependency or series of dependencies causes transaction A to appear to have executed before transaction B, but another series of dependencies causes B to appear before A. If that's the case, then the results can't be consistent with any serial execution of the transactions.

=== SSI Algorithm ===

Serializable transaction in PostgreSQL are implemented using
Serializable Snapshot Isolation (SSI), based on the work of Cahill,
et al. Fundamentally, this allows snapshot isolation to run as it
has, while monitoring for conditions which could create a serialization
anomaly.

SSI is based on the observation [2] that each snapshot isolation
anomaly corresponds to a cycle that contains a "dangerous structure"
of two adjacent rw-conflict edges:

Tin ----''rw''---> Tpivot ----''rw''---> Tout

SSI works by watching for this dangerous structure, and rolling back a transaction when needed to prevent any anomaly. This means it only needs to track rw-conflicts between concurrent transactions, not wr- and ww-dependencies. It also means there is a risk of false positives, because not every dangerous structure corresponds to an actual serialization failure.

The PostgreSQL implementation uses two additional optimizations:

# Tout must commit before any other transaction in the cycle (see proof of Theorem 2.1 of [2]). We only roll back a transaction if Tout commits before Tpivot and Tin.
# if Tin is read-only, there can only be an anomaly if Tout committed before Tin takes its snapshot. This optimization is an original one. Proof:
#* Because there is a cycle, there must be some transaction T0 that precedes Tin in the serial order. (T0 might be the same as Tout).
#* The dependency between T0 and Tin can't be a rw-conflict, because T1 was read-only, so it must be a ww- or wr-dependency. Those can only occur if T0 committed before T1 started.
#* Because Tout must commit before any other transaction in the cycle, it must commit before T0 commits -- and thus before T1 starts.

=== PostgreSQL Implementation ===

The implementation of serializable transactions for PostgreSQL is accomplished through Serializable Snapshot Isolation (SSI), based on the work of Cahill, et al[1][2]. Fundamentally, this allows snapshot isolation to run as it has, with monitoring for conditions which could create a serialization anomaly.

* Since this technique is based on Snapshot Isolation (SI), those areas in PostgreSQL which don't use SI can't be brought under SSI. This includes system tables, temporary tables, sequences, hint bit rewrites, etc. SSI can not eliminate existing anomalies in these areas.

* Any transaction which is run at a transaction isolation level other than SERIALIZABLE will not be affected by SSI. If you want to enforce business rules through SSI, all transactions should be run at the SERIALIZABLE transaction isolation level, and that should probably be set as the default.

* If all transactions are run at the SERIALIZABLE transaction isolation level, business rules can be enforced in triggers or application code without ever having a need to acquire an explicit lock or to use SELECT FOR SHARE or SELECT FOR UPDATE.

* Those who want to continue to use snapshot isolation without the additional protections of SSI (and the associated costs of enforcing those protections), can use the REPEATABLE READ transaction isolation level. This level will retain its legacy behavior, which is identical to the old SERIALIZABLE implementation and fully consistent with the standard's requirements for the REPEATABLE READ transaction isolation level.

* Performance under this SSI implementation will be significantly improved if transactions which don't modify permanent tables are declared to be READ ONLY before they begin reading data.

* Performance under SSI will tend to degrade more rapidly with a large number of active database transactions than under less strict isolation levels. Limiting the number of active transactions through use of a connection pool or similar techniques may be necessary to maintain good performance.

* Any transaction which must be rolled back to prevent serialization anomalies will fail with SQLSTATE 40001, which has a standard meaning of "serialization failure".

* This SSI implementation makes an effort to choose the transaction to be cancelled such that an immediate retry of the transaction can not fail due to conflicts with exactly the same transactions. Pursuant to this goal, no transaction is cancelled until one of the other transactions in the set of conflicts which could generate an anomaly has successfully committed. This is conceptually similar to how write conflicts are handled.

== Current Status ==

'''Accepted as a feature for PostgreSQL 9.1!'''

Many thanks to Joe, Heikki, Jeff, and Anssi for posing questions and making suggestions which have led to improvements in the patch! Thanks to Markus for providing dtester at a critical juncture, which allowed progress to continue, and Heikki for developing the src/test/isolation code to move the dcheck tests into the main PostgreSQL testing framework. Also, thanks to the many who have participated in discussions along the way.

There are some features which should be considered for 9.2 once 9.1 is settled down; most notably integration with hot standby and fine-grained support for index AMs other than btree. Most other proposed work is related to possible performance improvements, which should each be carefully benchmarked before being accepted. At the top of that list is better optimization of ''de facto'' read only transactions -- those which aren't flagged as read only, but which don't actually do any writes to permanent database tables.

== Development Path ==

In general, the approach taken was to try for the fastest possible implementation of a serializable isolation level which allowed no anomalies, even though it had many false positives and very poor performance, and then optimize until the rollback rate and overall performance were within a range which allows practical application. No existing isolation level was removed, since not everyone will want to pay the performance price for true serializable behavior. An important goal was that for those not using serializable transaction isolation, the patch doesn't cause performance regression.

=== Credits ===

'''Feature Authors''': [[User:Kgrittn|Kevin Grittner]] and [http://drkp.net/ Dan R. K. Ports].

'''Testing Support Authors''': Markus Wanner (dtester used during most of development) and Heikki Linnakangas (testing support consistent with other PostgreSQL regression testing, so that we had a testing suite suitable for commit).

'''Reviewers''': Joe Conway (warning elimination, bug chasing, and style comments), Jeff Davis (general review and found problems with GiST support and lack of 2PC support), Anssi Kääriäinen (found problems with conditional indexes and performance issue with sequential scans during testing with production data), YAMAMOTO Takashi (found numerous bugs during long and heavy testing), and Heikki Linnakangas (general review and many useful observations and suggestions, plus general improvements during commit process).

'''Committers''': Joe Conway (initial comment and name changes), Heikki Linnakangas (the bulk of the patch and most follow-up fixes), and Robert Haas (some follow-up fixes).

'''Thanks''' to all those who participated in the on-list discussions and offered advice and support off-list. There were so many who contributed in this way it would be practically impossible to generate an accurate list, but Robert Haas stands out for offering great advice on an overall development strategy.

'''Special thanks''' to Emmanuel Cecchet for pointing out the ACM SIGMOD paper in which this technique was originally published[1], and to all those at the University of Sidney who contributed to the development of this innovative technique. This is what turned the discussion from wrangling over how best to document existing behavior toward changing it.

=== Source Code Management ===

A "serializable" git branch has been set up at this location:

git://git.postgresql.org/git/users/kgrittn/postgres.git

http://git.postgresql.org/git/users/kgrittn/postgres.git

ssh://git@git.postgresql.org/users/kgrittn/postgres.git

http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=shortlog;h=refs/heads/serializable

=== Predicate Locking ===

Both S2PL and SSI require some form of predicate locking to handle situations where reads conflict with later inserts or with later updates which move data into the selected range. PostgreSQL didn't have predicate locking, so it needed to be added. Practical implementations of predicate locking generally involve acquiring locks against data as it is accessed, using multiple granularities (tuple, page, table, etc.) with escalation as needed to keep the lock count to a number which can be tracked within RAM structures. Coarse granularities can cause some false positive indications of conflict. The number of false positives can be influenced by plan choice.

==== Implementation overview ====

New RAM structures, inspired by those used to track traditional locks in PostgreSQL, but tailored to the needs of SIREAD predicate locking, will be used. These will refer to physical objects actually accessed in the course of executing the query, to model the predicates through inference. Anyone interested in this subject should review the Hellerstein, Stonebraker and Hamilton paper<nowiki>[3]</nowiki>, along with the locking papers referenced from that and the Cahill papers<nowiki>[1][2]</nowiki>.

Because the SIREAD locks don't block, traditional locking techniques must be modified. Intent locking (locking higher level objects before locking lower level objects) doesn't work with non-blocking "locks" (which are, in some respects, more like flags than locks).

A configurable amount of shared memory is reserved at postmaster start-up to track predicate locks. This size cannot be changed without a restart.
* To prevent resource exhaustion, multiple fine-grained locks may be promoted to a single coarser-grained lock as needed.
* An attempt to acquire an SIREAD lock on a tuple when the same transaction already holds an SIREAD lock on the page or the relation will be ignored. Likewise, an attempt to lock a page when the relation is locked will be ignored, and the acquisition of a coarser lock will result in the automatic release of all finer-grained locks it covers.

==== Heap locking ====

Predicate locks will be acquired for the heap based on the following:
* For a table scan, the entire relation will be locked.
* Each tuple read which is visible to the reading transaction will be locked, whether or not it meets selection criteria; except that there is no need to acquire an SIREAD lock on a tuple when the transaction already holds a write lock on any tuple representing the row, since a rw-dependency would also create a ww-dependency which has more aggressive enforcement and will thus prevent any anomaly.

==== Default index locking ====

There is a new ampredlocks flag in pg_am which should be set to false for any index which doesn't handle the predicate locking internally; indexes flagged this way will be predicate locked at the index relation level. Such a lock will conflict with any insert into the index, but will not conflict, for example, with deletes, HOT updates, or inserts which don't match the WHERE clause on an index (if present). This will allow correct behavior at the serializable transaction isolation level for new index types with minimal initial effort; but adding the predicate locking calls and changing the flag will improve performance in high contention workloads involving serializable transactions.

==== Index AM implementations ====

Since predicate locks only exist to detect writes which conflict with earlier reads, and heap tuple locks are acquired to cover all heap tuples actually read, including those read through indexes, the index tuples which were actually scanned are not of interest in themselves; we only care about their "new neighbors" -- later inserts into the index which ''would'' have been included in the scan had they existed at the time. Conceptually, we want to lock the ''gaps'' between and surrounding index entries within the scanned range.

''Correctness'' requires that any insert into an index generate a rw-conflict with a concurrent serializable transaction if, after that insert, re-execution of any index scan of the other transaction would access the heap for a row not accessed during the previous execution. Note that a non-HOT update which expires an old index entry covered by the scan and adds a new entry for the modified row's new tuple ''need not'' generate a conflict, although an update which "moves" a row into the scan ''must'' generate a conflict. While correctness allows false positives, they should be minimized for performance reasons.

Several optimizations are possible:

* An index scan which is just finding the right position for an index insertion or deletion need not acquire a predicate lock.

* An index scan which is comparing for equality on the entire key for a unique index need not acquire a predicate lock as long as a key is found corresponding to a visible tuple which has not been modified by another transaction -- there are no "between or around" gaps to cover.

* As long as built-in foreign key enforcement continues to use its current "special tricks" to deal with MVCC issues, predicate locks should not be needed for scans done by enforcement code.

* If a search determines that no rows can be found regardless of index contents because the search conditions are contradictory (e.g., x = 1 AND x = 2), then no predicate lock is needed.

Other index AM implementation considerations:

* If a btree search discovers that no root page has yet been created, a predicate lock on the index relation is required; otherwise btree searches must get to the leaf level to determine which tuples match, so predicate locks go there.

* GiST searches can determine that there are no matches at any level of the index, so there must be a predicate lock at each index level during a GiST search. An index insert at the leaf level can then be trusted to ripple up to all levels and locations where conflicting predicate locks may exist.

* The effects of page splits, overflows, consolidations, and removals must be carefully reviewed to ensure that predicate locks aren't "lost" during those operations, or kept with pages which could get re-used for different parts of the index.

=== Testing ===

For this development effort to succeed, it was absolutely necessary to have some client application which allowed execution of test scripts with specific interleaving of statements run against multiple backends. The dtester module from Markus Wanner was used for this during most of development. It requires python and several python packages (including twisted). Due to package dependencies and licensing issues the dtester module was not appropriate for commit to the PostgreSQL code base.

Heikki Linnakangas developed a testing framework based on existing regression test code which has been committed to src/test/isolation. Besides being compatible with other PostgreSQL testing, it runs faster than dtester. It doesn't provide a nice display of the results by statement ordering permutation, but that can be added if needed by filtering the current output.

Like many other proposed features and optimizations, this area could benefit from a "performance test farm" so that serializable performance can be better compared to other isolation levels, and so the performance impact of future enhancements can be determined.

=== Documentation ===

A README-SSI file was created, largely drawn from this Wiki page.

Someone with update rights to Wikipedia should probably update references there which will be outdated with this feature:

* http://en.wikipedia.org/wiki/Snapshot_isolation
* http://en.wikipedia.org/wiki/Isolation_%28database_systems%29

== Innovations ==

The PostgreSQL implementation of Serializable Snapshot Isolation differs from what is described in the cited papers for several reasons:
# PostgreSQL didn't have any existing predicate locking. It had to be added from scratch.
# The existing in-memory lock structures were not suitable for tracking SIREAD locks.
#* The database products used for the prototype implementations for the papers used update-in-place with a rollback log for their MVCC implementations, while PostgreSQL leaves the old version of a row in place and adds a new tuple to represent the row at a new location.
#* In PostgreSQL, tuple level locks are not held in RAM for any length of time; lock information is written to the tuples involved in the transactions.
#* In PostgreSQL, existing lock structures have pointers to memory which is related to a connection. SIREAD locks need to persist past the end of the originating transaction and even the connection which ran it.
#* PostgreSQL needs to be able to tolerate a large number of transactions executing while one long-running transaction stays open -- the in-RAM techniques discussed in the papers wouldn't support that.
# Unlike the database products used for the prototypes described in the papers, PostgreSQL didn't already have a true serializable isolation level distinct from snapshot isolation.
# PostgreSQL supports subtransactions -- an issue not mentioned in the papers.
# PostgreSQL doesn't assign a transaction number to a database transaction until and unless necessary.
# PostgreSQL has pluggable data types with user-definable operators, as well as pluggable index types, not all of which are based around data types which support ordering.
# Some possible optimizations became apparent during development and testing.

Differences from the implementation described in the papers are listed below.

* New structures needed to be created in shared memory to track the proper information for serializable transactions and their SIREAD locks.

* Because PostgreSQL does not have the same concept of an "oldest transaction ID" for all serializable transactions as assumed in the Cahill these, we track the oldest snapshot xmin among serializable transactions, and a count of how many active transactions use that xmin. When the count hits zero we find the new oldest xmin and run a clean-up based on that.

* Predicate locking in PostgreSQL will start at the tuple level when possible, with automatic conversion of multiple fine-grained locks to coarser granularity as need to avoid resource exhaustion. The amount of memory used for these structures will be configurable, to balance RAM usage against SIREAD lock granularity.

* A process-local copy of locks held by a process and the coarser covering locks with counts, are kept to support granularity promotion decisions with low CPU and locking overhead.

* Conflicts are identified by looking for predicate locks when tuples are written and looking at the MVCC information when tuples are read. There is no matching between two RAM-based locks.

* Because write locks are stored in the heap tuples rather than a RAM-based lock table, the optimization described in the Cahill thesis which eliminates an SIREAD lock where there is a write lock is implemented by the following:
*# When checking a heap write for conflicts against existing predicate locks, a tuple lock on the tuple being written is removed.
*# When acquiring a predicate lock on a heap tuple, we return quickly without doing anything if it is a tuple written by the reading transaction.

* Rather than using conflictIn and conflictOut pointers which use NULL to indicate no conflict and a self-reference to indicate multiple conflicts or conflicts with committed transactions, we use a list of rw-conflicts. With the more complete information, false positives are reduced and we have sufficient data for more aggressive clean-up and other optimizations.
** We can avoid ever rolling back a transaction until and unless there is a pivot where a transaction on the conflict *out* side of the pivot committed before either of the other transactions.
** We can avoid ever rolling back a transaction when the transaction on the conflict *in* side of the pivot is explicitly or implicitly READ ONLY unless the transaction on the conflict *out* side of the pivot committed before the READ ONLY transaction acquired its snapshot. (An implicit READ ONLY transaction is one which committed without writing, even though it was not explicitly declared to be READ ONLY.)
** We can more aggressively clean up conflicts, predicate locks, and SSI transaction information.

* Allow a READ ONLY transaction to "opt out" of SSI if there are no READ WRITE transactions which could cause the READ ONLY transaction to ever become part of a "dangerous structure" of overlapping transaction dependencies.

* Allow the user to request that a READ ONLY transaction ''wait'' until the conditions are right for it to start in the "opt out" state described above. We add a DEFERRABLE state to transactions, which is specified and maintained in a way similar to to READ ONLY. It is ignored for transactions which are not SERIALIZABLE ''and'' READ ONLY.

* When a transaction must be rolled back, we pick among the active transactions such that an immediate retry will not fail again on conflicts with the same transactions.

* We use the PostgreSQL SLRU system to hold summarized information about older committed transactions to put an upper bound on RAM used. Beyond that limit, information spills to disk. Performance can degrade in a pessimal situation, but it should be tolerable, and transactions won't need to be cancelled or blocked from starting.

== R&D Issues ==

This is intended to be the place to record specific issues which need more detailed review or analysis.

* '''WAL file replay'''. While serializable implementations using S2PL can guarantee that the write-ahead log contains commits in a sequence consistent with some serial execution of serializable transactions, SSI cannot make that guarantee. While the WAL replay is no less consistent than under snapshot isolation, it is possible that under PITR recovery or hot standby a database could reach a readable state where some transactions appear before other transactions which would have had to precede them to maintain serializable consistency. In essence, if we do nothing, WAL replay will be at snapshot isolation even for serializable transactions. Is this OK? If not, how do we address it?

* '''External replication'''. Look at how this impacts external replication solutions, like Postgres-R, Slony, pgpool, HS/SR, etc. This is related to the "WAL file replay" issue.

* '''UNIQUE btree search for equality on all columns'''. Since a search of a UNIQUE index using equality tests on all columns will lock the heap tuple if an entry is found, it appears that there is no need to get a predicate lock on the index in that case. A predicate lock ''is'' still needed for such a search if a matching index entry which points to a visible tuple is ''not'' found.

* '''Minimize touching of shared memory'''. Should lists in shared memory push entries which have just been returned to the ''front'' of the available list, so they will be popped back off soon and some memory might never be touched, or should we keep adding returned items to the ''end'' of the available list?

== Discussion ==

[http://archives.postgresql.org/message-id/4A0019EE.EE98.0025.0@wicourts.gov "Serializable Isolation without blocking" - discusses paper in ACM SIGMOD on SSI]

[http://archives.postgresql.org/message-id/4B2788EA020000250002D51C@gw.wicourts.gov "Update on true serializable techniques in MVCC" - discusses Cahill Doctoral Thesis on SSI]

[http://archives.postgresql.org/message-id/4B389C79020000250002D987@gw.wicourts.gov "Serializable implementation" - discusses Wisconsin Court System plans]

[http://archives.postgresql.org/message-id/4B3B88F4020000250002DAE1@gw.wicourts.gov "A third lock method" - discusses development path: rough prototype to refine toward production]

[http://archives.postgresql.org/message-id/1262718843.5908.183.camel@monkey-cat.sm.truviso.com "true serializability and predicate locking" - discusses GiST and GIN issues]

[http://archives.postgresql.org/message-id/4BF43DF702000025000318BE@gw.wicourts.gov WIP patch for serializable transactions with predicate locking]

[http://archives.postgresql.org/pgsql-hackers/2010-09/msg00022.php "serializable" in comments and names]

[http://archives.postgresql.org/message-id/4C8F5DB202000025000356A0@gw.wicourts.gov Serializable Snapshot Isolation]

[http://archives.postgresql.org/message-id/4CFB574702000025000382FD@gw.wicourts.gov serializable read only deferrable]

[http://archives.postgresql.org/pgsql-hackers/2010-12/msg02119.php SSI memory mitigation & false positive degradation]

== Presentations ==

From PostgreSQL Conference U.S. East 2010:
[[media:Transaction-Isolation-in-PostgreSQL.odp|Current Transaction Isolation in PostgreSQL and future directions]]

From PGCon 2011:
[http://drkp.net/drkp/papers/ssi-pgcon11-slides.pdf Serializable Snapshot Isolation: Making ISOLATION LEVEL SERIALIZABLE Provide Serializable Isolation]

== Publications ==

<nowiki>[1]</nowiki> [http://doi.acm.org/10.1145/1376616.1376690 Michael J. Cahill, Uwe Röhm, and Alan D. Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.] (This paper is listed mostly for context; the subsequent paper covers the same ground and more.)

<nowiki>[2]</nowiki> [http://hdl.handle.net/2123/5353 Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies.]

<nowiki>[3]</nowiki> [http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf Joseph M. Hellerstein, Michael Stonebraker and James Hamilton. 2007. Architecture of a Database System. Foundations and Trends(R) in Databases Vol. 1, No. 2 (2007) 141–259.]
Of particular interest:
* 6.1 A Note on ACID
* 6.2 A Brief Review of Serializability
* 6.3 Locking and Latching
* 6.3.1 Transaction Isolation Levels
* 6.5.3 Next-Key Locking: Physical Surrogates for Logical Properties

<nowiki>[4]</nowiki> [http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt SQL-92]
Search for ''serial execution'' to find the relevant section.