PostgreSQL wiki - User contributions [en]

PostgreSQL 9.3 Open Items

2013-05-19T16:36:59Z

Simon:

== Project Planning ==
See the [[PostgreSQL 9.3 Development Plan]].

== Blockers for 9.3 ==
* [http://www.postgresql.org/message-id/12365.1358098148@sss.pgh.pa.us Restore protection against accidentally creating stuff in pg_catalog schema]
** There are a couple of ways to fix that, per thread, but we need to do something.
* Improve the {{messageLink|m2zjwxpk8j.fsf@2ndQuadrant.fr|event trigger API documentation?}}
* [http://www.postgresql.org/message-id/5188CFFA.3020209@vmware.com Fast promotion failure] - doesn't appear to be an issue with fast promotion, not sure where though...

== Not Blockers for 9.3 ==

* Consider whether COPY into newly created tables should ALWAYS freeze, perhaps only when checksums enabled - requested by Noah, Robert
* [http://www.postgresql.org/message-id/CAHGQGwHs3r2k-N7C=vWEk5e-fE7sTGWgZjbkD6X2_s0h+zqoVQ@mail.gmail.com fast promotion and log_checkpoints] - trivial cosmetic issue; not a bug of any kind

== Meta-Issues ==

== Resolved Issues ==
* Restructure ProcessUtility to fix event-trigger clobber-cache-always failures
* Agree on the new page checksum algorithm
* Fix planner {{messageLink|6546.1365701142@sss.pgh.pa.us|equivalence-class bugs}}
* pg_ctl's new idempotent option [http://www.postgresql.org/message-id/CAMkU=1zKGzGoDoO=u4MON8h6Q=biRL59PTZvRmR9J7uX0yKoyA@mail.gmail.com broke crash recovery cases]
* [http://www.postgresql.org/message-id/CAHGQGwESOme9HUmUq_jTYi8j++qP2HoZxyqXR=37zuU8tHEOkw@mail.gmail.com VACUUM breaks matview scannability state]
* Do something about {{messageLink|14345.1365001149@sss.pgh.pa.us|unlogged matviews}}
* [http://www.postgresql.org/message-id/CAHGQGwEO1GF3=7LAeyn630+PNjjkC4fnwqx_L_Vyq6Y+OsB5jg@mail.gmail.com Another assertion failure at promotion]
* [http://www.postgresql.org/message-id/CAB7nPqRhuCuuD012GCB_tAAFrixx2WioN_zfXQcvLuRab8DN2g@mail.gmail.com Assertion failure when promoting node by deleting recovery.conf and restart node]

== Long-term Issues ==

[[Category:PostgreSQL 9.3]]

PostgreSQL 9.3 Open Items

2013-05-19T14:38:01Z

Simon: /* Not Blockers for 9.3 */

== Project Planning ==
See the [[PostgreSQL 9.3 Development Plan]].

== Blockers for 9.3 ==
* [http://www.postgresql.org/message-id/12365.1358098148@sss.pgh.pa.us Restore protection against accidentally creating stuff in pg_catalog schema]
** There are a couple of ways to fix that, per thread, but we need to do something.
* Improve the {{messageLink|m2zjwxpk8j.fsf@2ndQuadrant.fr|event trigger API documentation?}}
* [http://www.postgresql.org/message-id/CAB7nPqRhuCuuD012GCB_tAAFrixx2WioN_zfXQcvLuRab8DN2g@mail.gmail.com Assertion failure when promoting node by deleting recovery.conf and restart node]
* [http://www.postgresql.org/message-id/5188CFFA.3020209@vmware.com Fast promotion failure]

== Not Blockers for 9.3 ==

* Consider whether COPY into newly created tables should ALWAYS freeze, perhaps only when checksums enabled - requested by Noah, Robert
* [http://www.postgresql.org/message-id/CAHGQGwHs3r2k-N7C=vWEk5e-fE7sTGWgZjbkD6X2_s0h+zqoVQ@mail.gmail.com fast promotion and log_checkpoints] - trivial cosmetic issue; not a bug of any kind

== Meta-Issues ==

== Resolved Issues ==
* Restructure ProcessUtility to fix event-trigger clobber-cache-always failures
* Agree on the new page checksum algorithm
* Fix planner {{messageLink|6546.1365701142@sss.pgh.pa.us|equivalence-class bugs}}
* pg_ctl's new idempotent option [http://www.postgresql.org/message-id/CAMkU=1zKGzGoDoO=u4MON8h6Q=biRL59PTZvRmR9J7uX0yKoyA@mail.gmail.com broke crash recovery cases]
* [http://www.postgresql.org/message-id/CAHGQGwESOme9HUmUq_jTYi8j++qP2HoZxyqXR=37zuU8tHEOkw@mail.gmail.com VACUUM breaks matview scannability state]
* Do something about {{messageLink|14345.1365001149@sss.pgh.pa.us|unlogged matviews}}
* [http://www.postgresql.org/message-id/CAHGQGwEO1GF3=7LAeyn630+PNjjkC4fnwqx_L_Vyq6Y+OsB5jg@mail.gmail.com Another assertion failure at promotion]

== Long-term Issues ==

[[Category:PostgreSQL 9.3]]

PostgreSQL 9.3 Open Items

2013-05-19T14:37:19Z

Simon: /* Not Blockers for 9.3 */

== Project Planning ==
See the [[PostgreSQL 9.3 Development Plan]].

== Blockers for 9.3 ==
* [http://www.postgresql.org/message-id/12365.1358098148@sss.pgh.pa.us Restore protection against accidentally creating stuff in pg_catalog schema]
** There are a couple of ways to fix that, per thread, but we need to do something.
* Improve the {{messageLink|m2zjwxpk8j.fsf@2ndQuadrant.fr|event trigger API documentation?}}
* [http://www.postgresql.org/message-id/CAB7nPqRhuCuuD012GCB_tAAFrixx2WioN_zfXQcvLuRab8DN2g@mail.gmail.com Assertion failure when promoting node by deleting recovery.conf and restart node]
* [http://www.postgresql.org/message-id/5188CFFA.3020209@vmware.com Fast promotion failure]

== Not Blockers for 9.3 ==

* Consider whether message for COPY FREEZE needs to be added
* Consider whether COPY into newly created tables should ALWAYS freeze, perhaps only when checksums enabled
* [http://www.postgresql.org/message-id/CAHGQGwHs3r2k-N7C=vWEk5e-fE7sTGWgZjbkD6X2_s0h+zqoVQ@mail.gmail.com fast promotion and log_checkpoints] - trivial cosmetic issue; not a bug of any kind

== Meta-Issues ==

== Resolved Issues ==
* Restructure ProcessUtility to fix event-trigger clobber-cache-always failures
* Agree on the new page checksum algorithm
* Fix planner {{messageLink|6546.1365701142@sss.pgh.pa.us|equivalence-class bugs}}
* pg_ctl's new idempotent option [http://www.postgresql.org/message-id/CAMkU=1zKGzGoDoO=u4MON8h6Q=biRL59PTZvRmR9J7uX0yKoyA@mail.gmail.com broke crash recovery cases]
* [http://www.postgresql.org/message-id/CAHGQGwESOme9HUmUq_jTYi8j++qP2HoZxyqXR=37zuU8tHEOkw@mail.gmail.com VACUUM breaks matview scannability state]
* Do something about {{messageLink|14345.1365001149@sss.pgh.pa.us|unlogged matviews}}
* [http://www.postgresql.org/message-id/CAHGQGwEO1GF3=7LAeyn630+PNjjkC4fnwqx_L_Vyq6Y+OsB5jg@mail.gmail.com Another assertion failure at promotion]

== Long-term Issues ==

[[Category:PostgreSQL 9.3]]

BDR User Guide

2013-05-16T13:36:32Z

Simon: /* Replication of DML changes */

----
This page is the users and administrators guide for BDR. If you're looking for technical details on the project plan and implementation, see [[BDR Project]].
----

= BDR User Guide =

BDR (BiDrectional Replication) is a feature being developed for inclusion in PostgreSQL core that provides greatly enhanced replication capabilities.

BDR allows users to create a geographically distributed multi-master database using Logical Log Streaming Replication (LLSR) transport.
BDR is designed to provide both high availability and geographically distributed disaster recovery capabilities.

BDR is not “clustering” as some vendors use the term, in that it doesn't have a distributed lock manager, global transaction co-ordinator, etc. Each member server is separate yet connected, with design choices that allow separation between nodes that would not be possible with global transaction coordination.

Guidance on getting a testing setup established are in [[#Initial setup]]. Please read the full documentation if you intend to put BDR into production.

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows one PostgreSQL master (the "upstream master") to stream a sequence of changes to another read/write PostgreSQL server (the "downstream master"). Data is sent in one direction only over a normal libpq connection.

Multiple LLSR connections can be used to set up bi-directional replication as discussed later in this guide.

=== Overview of logical replication ===

In some ways LLSR is similar to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective; both replicate changes from one server to another. However, in LLSR the receiving server is also a full master database that can make changes, unlike the read-only replicas offered by PLSR hot standby. Additionally, LLSR is per-database, whereas PLSR is per-cluster and replicates all databases at once. There are many more differences discussed in the relevant sections of this document.

In LLSR the data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after <tt>CREATE DATABASE</tt>. A restart of the downstream master is also required. The upstream master only needs restarting if the <tt>max_logical_slots</tt> parameter is too low to allow a new replica to be added. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated. Setup is discussed in more detail below.

Changes are processed by the downstream master using <tt>bdr</tt> plug-ins. This allows flexible handing of replication input, including:

* BDR apply process - applies logical changes to the downstream master. The apply process makes changes directly rather than generating SQL text and then parse/plan/executing SQL.
* Textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* <tt>pg_xlogdump</tt> - examines physical WAL records and produces textual debugging output. This server program is included in PostgreSQL 9.3.

=== Replication of DML changes ===

All changes are replicated: <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> and <tt>TRUNCATE</tt>.

(TRUNCATE is not yet implemented, but will be implemented before the feature goes to final release).

Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though it has overheads that mean that it doesn't always use less bandwidth than PLSR.

Locks taken by <tt>LOCK</tt> and <tt>SELECT ... FOR UPDATE/SHARE</tt> on the upstream master are not replicated to downstream masters. Locks taken automatically by <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> or <tt>TRUNCATE</tt> *are* taken on the downstream master and may delay replication apply or concurrent transactions - see [[#Lock Conflicts|Lock Conflicts]].

<tt>TEMPORARY</tt> and <tt>UNLOGGED</tt> tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables. However, temporary tables remain specific to a particular session so creating a temporary table on the upstream master does not create a similar table on the downstream master.

<tt>DELETE</tt> and <tt>UPDATE</tt> statements that affect multiple rows on upstream master will cause a series of row changes on downstream master. These are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. <tt>UPDATE</tt>s and <tt>DELETE</tt>s require some form of unique constraint, either <tt>PRIMARY KEY</tt> or <tt>UNIQUE NOT NULL</tt>. A warning is issued in the downstream master's logs if the expected constraint is absent. <tt>INSERT</tt> on upstream master do not require a unique constraint in order to replicate correctly, though such usage would prevent conflict detection between multiple masters, if that was considered important.

<tt>UPDATE</tt>s that change the value of the Primary Key of a table will be replicated correctly.

The values applied are the final values from the <tt>UPDATE</tt> on the upstream master, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value. Volatile or stable functions are evaluated on the master side and the resulting values are replicated. Consequently any function side-effects (writing files, network socket activity, updating internal PostgreSQL variables, etc) will not occur on the replicas as the functions are not run again on the replica.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master (see "Limitations").

The current LLSR plugin implementation uses the binary libpq protocol, so it requires that the upstream and downstream master use same CPU architecture and word-length, i.e. "identical servers", as with physical replication. A textual output option will be added later for passing data between non-identical servers, e.g. laptops or mobile devices communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is efficiently implemented. Parallel apply is a possible future feature, especially for changes made while holding <tt>AccessExclusiveLock</tt>.

Changes are applied to the downstream master in the sequence in which they were commited on the upstream master. This is a known-good serialized ordering of changes, so replication serialization failures are not theoretically possible. Such failures are common in systems that use statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions spill to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

<tt>SET</tt> statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. We always update the correct tables, whatever the setting of <tt>search_path</tt>. Values are replicated correctly irrespective of the values of <tt>bytea_output</tt>, <tt>TimeZone</tt>, <tt>DateStyle</tt>, etc.

<tt>NOTIFY</tt> is not supported across log based replication, either physical or logical. <tt>NOTIFY</tt> and <tt>LISTEN</tt> will work fine on the upstream master but an upstream <tt>NOTIFY</tt> will not trigger a downstream <tt>LISTEN</tt>er.

In some cases, additional deadlocks can occur on apply. This causes an automatic retry of the apply of the replaying transaction and is only an issue if the deadlock recurs repeatedly, delaying replication.

From a performance and concurrency perspective the BDR apply process is similar to a normal backend. Frequent conflicts with locks from other transactions when replaying changes can slow things down and thus increase replication delay, so reducing the frequency of such conflicts can be a good way to speed things up. Any lock held by another transaction on the downstream master - <tt>LOCK</tt> statements, <tt>SELECT ... FOR UPDATE/FOR SHARE</tt>, or <tt>INSERT</tt>/<tt>UPDATE</tt>/<tt>DELETE</tt> row locks - can delay replication if the replication apply process needs to change the locked table/row.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching <tt>"Schemaname"."Tablename"</tt> on both upstream and downstream masters. e.g. changes from upstream's <tt>public.mytable</tt> will go to downstream's <tt>public.mytable</tt> while changes to the upstream <tt>mychema.mytable</tt> will go to the downstream <tt>myschema.mytable</tt>. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful synchronization of table definitions on each node otherwise <tt>ERROR</tt>s will be generated by the replication apply process. In general, tables must be an exact match between upstream and downstream masters.

There are no plans to implement working replication between dissimilar table definitions.

Tables must meet the following requirements to be compatible for purposes of LLSR:

* The downstream master must only have constraints (<tt>CHECK</tt>, <tt>UNIQUE</tt>, <tt>EXCLUSION</tt>, <tt>FOREIGN KEY</tt>, etc) that are also present on the upstream master. Replication may initially work with mismatched constraints but is likely to fail as soon as the downstream master rejects a row the upstream master accepted.
* The table referenced by a FOREIGN KEY on a downstream master must have all the keys present in the upstream master version of the same table.
* Storage parameters must match except for as allowed below
* Inheritance must be the same
* Dropped columns on master must be present on replicas
* Custom types and enum definitions must match exactly
* Composite types and enums must have the same oids on master and replication target
* Extensions defining types used in replicated tables must be of the same version or fully SQL-level compatible and the oids of the types they define must match.

The following differences are permissible between tables on different nodes:

* The table's <tt>pg_class</tt> oid, the oid of its associated TOAST table, and the oid of the table's rowtype in <tt>pg_type</tt> may differ;
* Extra or missing non-<tt>UNIQUE</tt> indexes
* Extra keys in downstream lookup tables for <tt>FOREIGN KEY</tt> references that are not present on the upstream master
* The table-level storage parameters for fillfactor and autovacuum
* Triggers and rules may differ (they are not executed by replication apply)

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR (see [[#LLSR Limitations|LLSR Limitations]]).

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of <tt>session_replication_role = origin</tt>.

In future it is expected that composite types and enums with non-identical oids will be converted using text output and input functions. This feature is not yet implemented.

=== LLSR limitations ===

The current LLSR implementation is subject to some limitations, which are being progressively removed as work progresses.

==== Data definition compatibility ====

Table definitions, types, extensions, etc must be near identical between upstream and downstream masters. See [[#Table definitions and DDL replication|Table definitions and DDL replication]].

==== DDL Replication ====

DDL replication is not yet supported.

==== Upstream feedback ====

No feedback from downstream masters to the upstream master is implemented for asynchronous LLSR, so upstream masters must be configured to keep enough WAL. See [[#Configuration|Configuration]].

==== TRUNCATE is not replicated ====

TRUNCATE is not yet supported, however workarounds with user-level triggers are possible and a ProcessUtility hook is planned to implement a similar approach globally.

The safest option is to define a user-level BEFORE trigger on each table that RAISEs an ERROR when TRUNCATE is attempted.

A simple truncate-blocking trigger is:

CREATE OR REPLACE FUNCTION deny_truncate() RETURNS trigger AS $$
BEGIN
IF tg_op = 'TRUNCATE' THEN
RAISE EXCEPTION 'TRUNCATE is not supported on this table. Please use DELETE FROM.';
ELSE
RAISE EXCEPTION 'This trigger only supports TRUNCATE';
END IF;
END;
$$ LANGUAGE plpgsql;

It can be applied to a table with:

CREATE TRIGGER deny_truncate_on_<tablename> BEFORE TRUNCATE ON <tablename>
FOR EACH STATEMENT EXECUTE PROCEDURE deny_truncate();

A PL/PgSQL DO block that queries <tt>pg_class</tt> and loops over it to <tt>EXECUTE</tt> a dynamic SQL <tt>CREATE TRIGGER</tt> command for each table that does not already have the trigger can be used to apply the trigger to all tables.

=== Initial setup ===

To set up LLSR or BDR you first need a patched PostgreSQL that can support LLSR/BDR, then you need to create one or more LLSR/BDR senders and one or more LLSR/BDR receivers.

==== Installing the patched PostgreSQL binaries ====

Currently BDR is only available in builds of the 'bdr' branch on Andres Freund's git repo on git.postgresql.org. PostgreSQL 9.2 and below do not support BDR, and 9.3 requires patches, so this guide will not work for you if you are trying to use a normal install of PostgreSQL.

First you need to clone, configure, compile and install like normal. Clone the sources from <tt>git://git.postgresql.org/git/users/andresfreund/postgres.git</tt> and checkout the <tt>bdr</tt> branch.

If you have an existing local PostgreSQL git tree specify it as <tt>--reference /path/to/existing/tree</tt> to greatly speed your git clone.

Example:

mkdir -p $HOME/bdr
cd bdr
git clone git://git.postgresql.org/git/users/andresfreund/postgres.git $HOME/bdr/postgres-bdr-src
cd postgres-bdr-src
./configure --prefix=$HOME/bdr/postgres-bdr-bin
make install
cd contrib/bdr
make install

This will put everything in <tt>$HOME/bdr</tt>, with the source code and build tree in <tt>$HOME/bdr/postgres-bdr-src</tt> and the installed PostgreSQL in <tt>$HOME/bdr/postgres-bdr-bin</tt>. This is a convenient setup for testing and development because it doesn't require you to set up new users, wrangle permissions, run anything as root, etc, but it isn't recommended that you deploy this way in production.

To actually use these new binaries you will need to:

export PATH=$HOME/bdr/postgres-bdr-bin/bin:$PATH

before running <tt>initdb</tt>, <tt>postgres</tt>, etc. You don't have to use the <tt>psql</tt> or <tt>libpq</tt> you compiled but you're likely to get version mismatch warnings if you don't.

=== Parameter Reference ===

The following parameters are new or have been changed in PostgreSQL's new logical streaming replication.

==== <tt>shared_preload_libraries = ‘bdr’</tt> ====

To load support for receiving changes on a downstream master, the <tt>bdr</tt> library must be added to the existing ‘shared_preload_libraries’ parameter. This loads the bdr library during postmaster start-up and allows it to create the required background worker(s).

Upstream masters don't need to load the bdr library unless they're also operating as a downstream master as is the case in a BDR configuration.

==== <tt>bdr.connections</tt> ====

A comma-separated list of upstream master connection names is specified in <tt>bdr.connections</tt>. These names must be simple alphanumeric strings. They are used when naming the connection in error messages, configuration options and logs, but are otherwise of no special meaning.

A typical two-upstream-master setting might be:

bdr.connections = ‘upstream1, upstream2’

==== <tt>bdr.<connection_name>.dsn</tt> ====

Each connection name must have at least a data source name specified using the <tt>bdr.<connection_name>.dsn</tt> parameter. The DSN syntax is the same as that used by libpq so it is not discussed in further detail here. A <tt>dbname</tt> for the database to connect to must be specified; all other parts of the DSN are optional.

The local (downstream) database name is assumed to be the same as the name of the upstream database being connected to, though future versions will make this configurable.

For the above two-master setting for <tt>bdr.connections</tt> the DSNs might look like:

bdr.upstream1.dsn = 'host=10.1.1.2 user=postgres dbname=replicated_db'
bdr.upstream2.dsn = 'host=10.1.1.3 user=postgres dbname=replicated_db'

==== <tt>max_logical_slots</tt> ====

The new parameter <tt>max_logical_slots</tt> has been added for use on both upstream and downstream masters. This parameter controls the maximum number of logical replication slots - upstream or downstream - that this cluster may have at a time. It must be set at postmaster start time.

As logical replication slots are persistent, slots are consumed even by replicas that are not currently connected. Slot management is discussed in Starting, Stopping and Managing Replication.

<tt>max_logical_slots</tt> should be set to the sum of the number of logical replication upstream masters this server will have plus the number of logical replication downstream masters will connect to it it.

==== <tt>wal_level = 'logical'</tt> ====

A new setting, <tt>'logical'</tt>, has been added for the existing <tt>wal_level</tt> parameter. <tt>‘logical’</tt> includes everything that the existing <tt>hot_standby</tt> setting does and adds additional details required for logical changeset decoding to the write-ahead logs.

This additional information is consumed by the upstream-master-side xlog decoding worker. Downstream masters that do not also act as upstream masters do not require <tt>wal_level</tt> to be increased above the default <tt>'minimal'</tt>.

<tt>wal_level</tt>, except for the new <tt>'logical'</tt> setting, is [http://www.postgresql.org/docs/current/static/runtime-config-wal.html documented in the main PostgreSQL manual].

==== <tt>max_wal_senders</tt> ====

Logical replication hasn't altered the <tt>max_wal_senders</tt> parameter, but it is important in upstream masters for logical replication and BDR because every logical sender consumes a <tt>max_wal_senders</tt> entry.

You should configure <tt>max_wal_senders</tt> to the sum of the number of physical and logical replicas you want to allow an upstream master to serve. If you intend to use <tt>pg_basebackup</tt> you should add at least two more senders to allow for its use.

Like <tt>max_logical_slots</tt>, <tt>max_wal_senders</tt> entries don't cost a large amount of memory, so you can overestimate fairly safely.

<tt>max_wal_senders</tt> is documented in [http://www.postgresql.org/docs/current/static/runtime-config-replication.html the main PostgreSQL documentation].

==== <tt>wal_keep_segments</tt> ====

Like <tt>max_wal_senders</tt>, the <tt>wal_keep_segments</tt> parameter isn't directly changed by logical replication but is still important for upstream masters. It is not required on downstream-only masters.

<tt>wal_keep_segments</tt> should be set to a value that allows for some downtime or unreachable periods for downstream masters and for heavy bursts of write activity on the upstream master.

Keep in mind that enough disk space must be available for the WAL segments, each of which is 16MB. If you run out of disk space the server will halt until disk space is freed and it may be quite difficult to free space when you can no longer start the server.

If you exceed the required <tt>wal_keep_segments</tt> and "Insufficient WAL segments retained" error will be reported. See [[#Troubleshooting|Troubleshooting]].

<tt>wal_keep_segments</tt> is documented in the [http://www.postgresql.org/docs/current/static/runtime-config-replication.html the main PostgreSQL manual].

==== <tt>track_commit_timestamp</tt> ====

Setting this parameter to "on" enables commit timestamp tracking, which is used to implement last-UPDATE-wins conflict resolution.

=== Configuration ===

Details on individual parameters are described in the [[parameter reference]] section.

The following configuration is an example of a simple one-way LLSR replication setup - a single upstream master to a single downstream master.

The upstream master (sender)'s <tt>postgresql.conf</tt> should contain settings like:

wal_level = 'logical' # Include enough info for logical replication
max_logical_slots = X # Number of LLSR senders + any receivers
max_wal_senders = Y # Y = max_logical_slots plus any physical
# streaming requirements
wal_keep_segments = 5000 # Master must retain enough WAL segments to let
# replicas catch up. Correct value depends on
# rate of writes on master, max replica downtime
# allowable. 5000 segments requires 78GB
# in pg_xlog

Downstream (receiver) <tt>postgresql.conf</tt>:

shared_preload_libraries = 'bdr'

bdr.connections="name_of_upstream_master" # list of upstream master nodenames
bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection
# from downstream to upstream master
bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case
# where the databasename on upstream
# and downstream master differ.
# (Not yet implemented)
bdr.<nodename>.apply_delay # optional parameter to delay apply of
# transactions, time in milliseconds
bdr.synchronous_commit = ...; # optional parameter to set the
# synchronous_commit parameter the
# apply processes will be using
max_logical_slots = X # set to the number of remotes

Note that a server can be both sender and receiver, either two servers to each other or more complex configurations like replication chains/trees.

The upstream (sender) <tt>pg_hba.conf</tt> must be configured to allow the downstream master to connect for replication. Otherwise you'll see errors like the following on the downstream master:

FATAL: could not connect to the primary server: FATAL: no pg_hba.conf entry for replication connection from host "[local]", user "postgres"

A suitable <tt>pg_hba.conf</tt> entry for a replication connection from the replica server 10.1.4.8 might be:

host replication postgres 10.1.4.8/32 trust

(the user name should match the user name configured in the downstream master's dsn. md5 password authentication is supported.)

For more details on these parameters, see [[#Parameter Reference|Parameter Reference]].

=== Troubleshooting ===

==== Could not access file "bdr": No such file or directory ====

If you see the error:

FATAL: could not access file "bdr": No such file or directory

when starting a database set up to receive BDR replication, you probably forgot to install <tt>contrib/bdr</tt>. See above.

==== Invalid value for parameter ====

An error like:

LOG: invalid value for parameter ...

when setting one of these parameters means your server doesn't support logical replication and will need to be patched or updated.

==== Insufficient WAL segments retained ("requested WAL segment ... has already been removed") ====

If <tt>wal_keep_segments</tt> is insufficient to meet the requirements of a replica that has fallen far behind, the master will report errors like:

ERROR: requested WAL segment 00000001000000010000002D has already been removed

Currently the replica errors look like:

WARNING: Starting logical replication
LOG: data stream ended
LOG: worker process: master (PID 23812) exited with exit code 0
LOG: starting background worker process "master"
LOG: master initialized on master, remote dbname=master port=5434 replication=true fallback_application_name=bdr
LOG: local sysid 5873181566046043070, remote: 5873181102189050714
LOG: found valid replication identifier 1
LOG: starting up replication at 1 from 1/2D9CA220

but a more explicit error message for this condition is planned.

The only way to recover from this fault is to re-seed the replica database.

This fault could be prevented with feedback from the replica to the master, but this feature is not planned for the first release of BDR. Another alternative considered for future releases is making wal_keep_segments a dynamic parameter that is sized on demand.

Monitoring of maximum replica lag and appropriate adjustment of wal_keep_segments will prevent this fault from arising.

==== Couldn't find logical slot ====

An error like:

ERROR: couldn't find logical slot "bdr: 16384:5873181566046043070-1-24596:"

on the upstream master suggests that a downstream master is trying to connect to a logical replication slot that no longer exists. The slot can not be re-created, so it is necessary to re-seed the downstream replica database.

=== Operational Issues and Debugging ===

In LLSR there are no user-level (ie SQL visible) ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

The following views are available for monitoring replication activity:

* <tt>[http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE pg_stat_replication]</tt>
* <tt>pg_stat_logical_replication</tt> (described below)
* <tt>pg_stat_bdr</tt> (described below)

The following configuration and logging parameters are useful for monitoring replication:

* <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt>

==== pg_stat_logical_replication ====

The new <tt>pg_stat_logical_replication</tt> view is specific to logical replication. It is based on the underlying <tt>pg_stat_get_logical_replication_slots</tt> function and has the following structure:

View "pg_catalog.pg_stat_logical_replication"
Column | Type | Modifiers
--------------------------+---------+-----------
slot_name | text |
plugin | text |
database | oid |
active | boolean |
xmin | xid |
last_required_checkpoint | text |

It contains one row for every connection from a downstream master to the server being queried (the upstream master). On a standalone PostgreSQL server or a downstream-only master this view will contain no rows.

* <tt>slot_name</tt>: An internal name for a given logical replication slot (a connection from a downstream master to this upstream master). This slot name is used by the downstream master to uniquely identify its self and is used with the <tt>pg_receivellog</tt> command when managing logical replication slots. The slot name is composed of the decoding plugin name, the upstream database oid, the downstream system identifier (from <tt>pg_control</tt>), the downstream slot number, and the downstream database oid.

* <tt>plugin</tt>: The logical replication plugin being used to decode WAL archives. You'll generally only see <tt>bdr_output</tt> here.

* <tt>database</tt>: The oid of the database being replicated by this slot. You can get the database name by joining on <tt>pg_database.oid</tt>.

* <tt>active</tt>: Whether this slot currently has an active connection.

* <tt>xmin</tt>: The lowest transaction ID this replication slot can "see", like the xmin of a transaction or prepared transaction. xmin should keep on advancing as replication continues.

* <tt>last_required_checkpoint</tt>: The checkpoint identifying the oldest WAL record required to bring this slot up to date with the upstream master. (This column is likely to be removed in a future version).

==== pg_stat_bdr ====

The <tt>pg_stat_bdr</tt> view is supplied by the <tt>bdr</tt> extension. It provides information on a server's connection(s) to its upstream master(s). It is not present on upstream-only masters.

The primary purpose of this view is to report statistics on the progress of LLSR apply on a per-upstream master connection basis.

View structure:

View "public.pg_stat_bdr"
Column | Type | Modifiers
--------------------+--------+-----------
rep_node_id | oid |
riremotesysid | name |
riremotedb | oid |
rilocaldb | oid |
nr_commit | bigint |
nr_rollback | bigint |
nr_insert | bigint |
nr_insert_conflict | bigint |
nr_update | bigint |
nr_update_conflict | bigint |
nr_delete | bigint |
nr_delete_conflict | bigint |
nr_disconnect | bigint |

Fields:

* <tt>rep_node_id</tt>: An internal identifier for the replication slot.

* <tt>riremotesysid</tt>: The remote database system identifier, as reported by the <tt>Database system identifier</tt> line of <tt>pg_controldata /path/to/datadir</tt>

* <tt>riremotedb</tt>: The remote database OID, ie the <tt>oid</tt> column of the remote server's <tt>pg_catalog.pg_database</tt> entry for the replicated database. You can get the database name with <tt>select datname from pg_database where oid = 12345</tt> (where '12345' is the <tt>riremotedb</tt> oid).

* <tt>rilocaldb </tt>: The local database OID, with the same meaning as <tt>riremotedb</tt> but with oids from the local system.

''The rest of the rows are statistics about this upstream master slot'':

* <tt>nr_commit</tt>: Number of commits applied to date from this master

* <tt>nr_rollback</tt>: Number of rollbacks performed by this apply process due to recoverable errors (deadlock retries, lost races, etc) or unrecoverable errors like mismatched constraint errors.

* <tt>nr_insert</tt>: Number of <tt>INSERT</tt>s performed

* <tt>nr_insert_conflict</tt>: Number of <tt>INSERT</tt>s that resulted in conflicts.

* <tt>nr_update</tt>: Number of <tt>UPDATE</tt>s performed

* <tt>nr_update_conflict</tt>: Number of <tt>UPDATE</tt>s that resulted in conflicts.

* <tt>nr_delete</tt>: Number of deletes performed

* <tt>nr_delete_conflict</tt>: Number of deletes that resulted in conflicts.

* <tt>nr_disconnect</tt>: Number of times this apply process has lost its connection to the upstream master since it was started.

This view does not contain any information about how far behind the upstream master this downstream master is. The upstream master's <tt>pg_stat_logical_replication</tt> and <tt>pg_stat_replication</tt> views must be queried to determine replication lag.

==== Monitoring replication status and lag ====

As with any replication setup, it is vital to monitor replication status on all BDR nodes to ensure no node is lagging severely behind the others or is stuck.

In the case of BDR a stuck or crashed node will eventually cause disk space and table bloat problems on other masters so stuck nodes should be detected and removed or repaired in a reasonably timely manner. Exactly how urgent this is depends on the workload of the BDR group.

The <tt>pg_stat_logical_replication</tt> view described above may be used to verify that a downstream master is connected to its upstream master - the <tt>active</tt> boolean column is <tt>t</tt> if there's a downstream master connected.

The <tt>xmin</tt> column provides an indication of whether replication is advancing; it should increase as replication progresses. There is no simple way to turn <tt>xmin</tt> into the time the last applied transaction was committed on the master, so it doesn't provide an indication of wall-clock lag.

To determine wall-clock replication lag an application-level ticker may be used to periodically update a timestamp in a replicated table. The difference between this timestamp on the upstream and downstream masters provides the wall-clock replication lag. For BDR one row may be added to the table for each BDR master, giving an indication of how much lag each master has relative to each other master.

=== Table and index usage statistics ===

Statistics on table and index usage are updated normally by the downstream master. This is essential for correct function of auto-vacuum. If there are no local writes on the downstream master and stats have not been reset these two views should show matching results between upstream and downstream:

* <tt>pg_stat_user_tables</tt>
* <tt>pg_statio_user_tables</tt>

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform <tt>UPDATE</tt>s and <tt>DELETE</tt>s than non-identifying indexes are.

The built-in index monitoring views are:

* <tt>pg_stat_user_indexes</tt>
* <tt>pg_statio_user_indexes</tt>

All these views are discussed in [http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE the PostgreSQL documentation on the statistics views].

=== Starting, stopping and managing replication ===

Replication is managed with the <tt>postgresql.conf</tt> settings described in "Parameter Reference" and "Configuration" above, and using the <tt>pg_receivellog</tt> utility command.

==== Starting a new LLSR connection ====

Logical replication is started automatically when a database is configured as a downstream master in <tt>postgresql.conf</tt> (see [[#Configuration|Configuration]]) and the postmaster is started. No explicit action is required to start replication, but replication will not actually work unless the upstream and downstream databases are identical within the requirements set by LLSR in the [[#Table definitions and DDL replication||Table definitions and DDL replication]] section.

<tt>pg_dump</tt> and <tt>pg_restore</tt> may be used to set up the new replica's database.

==== Viewing logical replication slots ====

Examining the state of logical replication is discussed in [[#Monitoring|Monitoring]].

==== Temporarily stopping an LLSR replica ====

LLSR replicas can be temporarily stopped by shutting down the downstream master's postmaster.

If the replica is not started back up before the upstream master discards the oldest WAL segment required for the downstream master to resume replay, as identified by the <tt>last_required_checkpoint</tt> column of <tt>pg_catalog.pg_stat_logical_replication</tt> then the replica will not resume replay. The error [[#Insufficient_WAL_segments_retained_.28.22requested_WAL_segment_..._has_already_been_removed.22.29|Insufficient WAL segments retained]] will be reported in the upstream master's logs. The replica must be re-created for replication to continue.

==== Removing an LLSR replica permanently ====

To remove a replication connection permanently, remove its entries from the downstream master's <tt>postgresql.conf</tt>, restart the downstream master, then use <tt>pg_receivellog</tt> to remove the replication slot on the upstream master.

It is important to remove the replication slot from the upstream master(s) to prevent xid wrap-around problems and issues with table bloat caused by delayed vacuum.

==== Cleaning up abandoned replication slots ====

To remove a replication slot that was used for a now-defunct replica, find its slot name from the <tt>[[#pg_stat_logical_replication|pg_stat_logical_replication]]</tt> view on the upstream master then run:

pg_receivellog -p 5434 -h master-hostname -d dbname \
--slot='bdr: 16384:5873181566046043070-1-16384:' --stop

where the argument to '--slot' is the slot name you found from the view.

You may need to do this if you've created and then deleted several replicas so <tt>max_logical_slots</tt> has filled up with entries for replicas that no longer exist.

== Bi-Directional Replication ==

Bi-Directional replication is built directly on LLSR by configuring two or more servers as both upstream ''and'' downstream masters of each other.

All of the Log Level Streaming Replication documentation applies to BDR and should be read before moving on to reading about and configuring BDR.

=== Bi-Directional Replication Use Cases ===

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

==== Simple multi-master pair ====

A simple mulit-master "HA Cluster" with two servers:

* Server "Alpha" - Master
* Server "Bravo" - Master

===== Configuration =====

Alpha:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="bravo"
bdr.bravo.dsn = 'dbname=dbtoreplicate'
track_commit_timestamp = on

Bravo:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="alpha"
bdr.alpha.dsn = 'dbname=dbtoreplicate'
track_commit_timestamp = on

See [[#Configuration|Configuration]] for an explanation of these parameters.

==== HA and Logical Standby ====
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

"HA Cluster":

* Server "Alpha" - Current Master
* Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
* Server "Charlie" - "Logical Standby" - downstream master

==== Very High Availability Multi-Master ====
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

==== 3-remote site simple Multi-Master Plex ====

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Alpha, Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Charlie using logical streaming replication

===== Configuration =====

If you wanted to test this configuration locally you could run three PostgreSQL instances on different ports. Such a configuration would look like the following if the port numbers were used as node names for the sake of notational clarity:

Config for node_5440:

port = 5440
bdr.connections='node_5441,node_5442'
bdr.node_5441.dsn='port=5441 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5440,node_5442'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440,node_5441'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5441.dsn='port=5441 dbname=postgres'

In a typical real-world configuration each server would be on the same port on a different host instead.

==== 3-remote site simple Multi-Master Circular Replication ====

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases. It's also less resilient to network disruptions and node faults.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

TODO: Regrettably this doesn't actually work yet because we don't cascade logical changes (yet).

===== Configuration =====

Using node names that match port numbers, for clarity

Config for node_5440:

port = 5440
bdr.connections='node_5441'
bdr.node_5441.dsn='port=5441 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5442'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440'
bdr.node_5440.dsn='port=5440 dbname=postgres'

This would usually be done in the real world with databases on different hosts, all running on the same port.

==== 3-remote site Max Availability Multi-Master Plex ====

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

Config left as an exercise for the reader.

==== N-site symmetric cluster replication ====

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

==== Complex/Assymetric Replication ====

Variety of options are possible.

=== Conflict Avoidance ===

==== Distributed Locking ====

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications as very low latency is critical for acceptable performance.

Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible but allow some types of conflict to occur and and resolve them when they arise.

==== Global Sequences ====

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

The SQL standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using <tt>DEFAULT nextval('mysequence')</tt>, as with PostgreSQL's <tt>SERIAL</tt> pseudo-type.

BDR requires sequences to work together across multiple nodes. This is implemented as a new <tt>SequenceAccessMethod</tt> API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

=== Conflict Detection & Resolution ===

Because local writes can occur on a master, conflict detection and avoidance is a concern for basic LLSR setups as well as full BDR configurations.

==== Lock Conflicts ====

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt> facility to look for issues with apply blocking on locks.

By concurrent activity on a row, we include

* explicit row level locking (<tt>SELECT ... FOR UPDATE/FOR SHARE</tt>)
* locking from foreign keys
* implicit locking because of row <tt>UPDATE</tt>s, <tt>INSERT</tt>s or <tt>DELETE</tt>s, either from local activity or apply from other servers

==== Data Conflicts ====

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in a consistent and idempotent manner so that all servers end up with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from <tt>pg_control</tt> though this may change in a future release.

<tt>UPDATE</tt>s and <tt>INSERT</tt>s may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur. The errors causing the conflict can be seen in the error log of the downstream master with the problem.

Updates which cannot locate a row are presumed to be <tt>DELETE</tt>/<tt>UPDATE</tt> conflicts. These are accepted as successful operations but in the case of <tt>UPDATE</tt> the data in the <tt>UPDATE</tt> is discarded.

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins. It is not practical to decide when a row should be merged and when a last-update-wins stragegy should be used at the database level; such decision making would require support for application-specific conflict resolution plugins.

Changing unlogged and logged tables in the same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

==== Examples ====

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-05-16T13:32:57Z

Simon: /* Replication of DML changes */

----
This page is the users and administrators guide for BDR. If you're looking for technical details on the project plan and implementation, see [[BDR Project]].
----

= BDR User Guide =

BDR (BiDrectional Replication) is a feature being developed for inclusion in PostgreSQL core that provides greatly enhanced replication capabilities.

BDR allows users to create a geographically distributed multi-master database using Logical Log Streaming Replication (LLSR) transport.
BDR is designed to provide both high availability and geographically distributed disaster recovery capabilities.

BDR is not “clustering” as some vendors use the term, in that it doesn't have a distributed lock manager, global transaction co-ordinator, etc. Each member server is separate yet connected, with design choices that allow separation between nodes that would not be possible with global transaction coordination.

Guidance on getting a testing setup established are in [[#Initial setup]]. Please read the full documentation if you intend to put BDR into production.

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows one PostgreSQL master (the "upstream master") to stream a sequence of changes to another read/write PostgreSQL server (the "downstream master"). Data is sent in one direction only over a normal libpq connection.

Multiple LLSR connections can be used to set up bi-directional replication as discussed later in this guide.

=== Overview of logical replication ===

In some ways LLSR is similar to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective; both replicate changes from one server to another. However, in LLSR the receiving server is also a full master database that can make changes, unlike the read-only replicas offered by PLSR hot standby. Additionally, LLSR is per-database, whereas PLSR is per-cluster and replicates all databases at once. There are many more differences discussed in the relevant sections of this document.

In LLSR the data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after <tt>CREATE DATABASE</tt>. A restart of the downstream master is also required. The upstream master only needs restarting if the <tt>max_logical_slots</tt> parameter is too low to allow a new replica to be added. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated. Setup is discussed in more detail below.

Changes are processed by the downstream master using <tt>bdr</tt> plug-ins. This allows flexible handing of replication input, including:

* BDR apply process - applies logical changes to the downstream master. The apply process makes changes directly rather than generating SQL text and then parse/plan/executing SQL.
* Textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* <tt>pg_xlogdump</tt> - examines physical WAL records and produces textual debugging output. This server program is included in PostgreSQL 9.3.

=== Replication of DML changes ===

All changes are replicated: <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> and <tt>TRUNCATE</tt>.

(TRUNCATE is not yet implemented, but will be implemented before the feature goes to final release).

Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though it has overheads that mean that it doesn't always use less bandwidth than PLSR.

Locks taken by <tt>LOCK</tt> and <tt>SELECT ... FOR UPDATE/SHARE</tt> on the upstream master are not replicated to downstream masters. Locks taken automatically by <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> or <tt>TRUNCATE</tt> *are* taken on the downstream master and may delay replication apply or concurrent transactions - see [[#Lock Conflicts|Lock Conflicts]].

<tt>TEMPORARY</tt> and <tt>UNLOGGED</tt> tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables. However, temporary tables remain specific to a particular session so creating a temporary table on the upstream master does not create a similar table on the downstream master.

<tt>DELETE</tt> and <tt>UPDATE</tt> statements that affect multiple rows on upstream master will cause a series of row changes on downstream master. These are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. <tt>UPDATE</tt>s and <tt>DELETE</tt>s require some form of unique constraint, either <tt>PRIMARY KEY</tt> or <tt>UNIQUE NOT NULL</tt>. A warning is issued in the downstream master's logs if the expected constraint is absent. <tt>INSERT</tt> on upstream master do not require a unique constraint in order to replicate correctly, though such usage would prevent conflict detection between multiple masters, if that was considered important.

<tt>UPDATE</tt>s that change the value of the Primary Key of a table will be replicated correctly.

The values applied are the final values from the <tt>UPDATE</tt> on the upstream master, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value. Volatile or stable functions are evaluated on the master side and the resulting values are replicated. Consequently any function side-effects (writing files, network socket activity, updating internal PostgreSQL variables, etc) will not occur on the replicas as the functions are not run again on the replica.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master (see "Limitations").

The current LLSR plugin implementation uses the binary libpq protocol, so it requires that the upstream and downstream master use same CPU architecture and word-length, i.e. "identical servers", as with physical replication. A textual output option will be added later for passing data between non-identical servers, e.g. laptops or mobile devices communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is efficiently implemented. Parallel apply is a possible future feature, especially for changes made while holding <tt>AccessExclusiveLock</tt>.

Changes are applied to the downstream master in the sequence in which they were commited on the upstream master. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions spill to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

<tt>SET</tt> statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. We always update the correct tables, whatever the setting of <tt>search_path</tt>. Values are replicated correctly irrespective of the values of <tt>bytea_output</tt>, <tt>TimeZone</tt>, <tt>DateStyle</tt>, etc.

<tt>NOTIFY</tt> is not supported across log based replication, either physical or logical. <tt>NOTIFY</tt> and <tt>LISTEN</tt> will work fine on the upstream master but an upstream <tt>NOTIFY</tt> will not trigger a downstream <tt>LISTEN</tt>er.

In some cases, additional deadlocks can occur on apply. This causes an automatic retry of the apply of the replaying transaction and is only an issue if the deadlock recurs repeatedly, delaying replication.

From a performance and concurrency perspective the BDR apply process is similar to a normal backend. Frequent conflicts with locks from other transactions when replaying changes can slow things down and thus increase replication delay, so reducing the frequency of such conflicts can be a good way to speed things up. Any lock held by another transaction on the downstream master - <tt>LOCK</tt> statements, <tt>SELECT ... FOR UPDATE/FOR SHARE</tt>, or <tt>INSERT</tt>/<tt>UPDATE</tt>/<tt>DELETE</tt> row locks - can delay replication if the replication apply process needs to change the locked table/row.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching <tt>"Schemaname"."Tablename"</tt> on both upstream and downstream masters. e.g. changes from upstream's <tt>public.mytable</tt> will go to downstream's <tt>public.mytable</tt> while changes to the upstream <tt>mychema.mytable</tt> will go to the downstream <tt>myschema.mytable</tt>. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful synchronization of table definitions on each node otherwise <tt>ERROR</tt>s will be generated by the replication apply process. In general, tables must be an exact match between upstream and downstream masters.

There are no plans to implement working replication between dissimilar table definitions.

Tables must meet the following requirements to be compatible for purposes of LLSR:

* The downstream master must only have constraints (<tt>CHECK</tt>, <tt>UNIQUE</tt>, <tt>EXCLUSION</tt>, <tt>FOREIGN KEY</tt>, etc) that are also present on the upstream master. Replication may initially work with mismatched constraints but is likely to fail as soon as the downstream master rejects a row the upstream master accepted.
* The table referenced by a FOREIGN KEY on a downstream master must have all the keys present in the upstream master version of the same table.
* Storage parameters must match except for as allowed below
* Inheritance must be the same
* Dropped columns on master must be present on replicas
* Custom types and enum definitions must match exactly
* Composite types and enums must have the same oids on master and replication target
* Extensions defining types used in replicated tables must be of the same version or fully SQL-level compatible and the oids of the types they define must match.

The following differences are permissible between tables on different nodes:

* The table's <tt>pg_class</tt> oid, the oid of its associated TOAST table, and the oid of the table's rowtype in <tt>pg_type</tt> may differ;
* Extra or missing non-<tt>UNIQUE</tt> indexes
* Extra keys in downstream lookup tables for <tt>FOREIGN KEY</tt> references that are not present on the upstream master
* The table-level storage parameters for fillfactor and autovacuum
* Triggers and rules may differ (they are not executed by replication apply)

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR (see [[#LLSR Limitations|LLSR Limitations]]).

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of <tt>session_replication_role = origin</tt>.

In future it is expected that composite types and enums with non-identical oids will be converted using text output and input functions. This feature is not yet implemented.

=== LLSR limitations ===

The current LLSR implementation is subject to some limitations, which are being progressively removed as work progresses.

==== Data definition compatibility ====

Table definitions, types, extensions, etc must be near identical between upstream and downstream masters. See [[#Table definitions and DDL replication|Table definitions and DDL replication]].

==== DDL Replication ====

DDL replication is not yet supported.

==== Upstream feedback ====

No feedback from downstream masters to the upstream master is implemented for asynchronous LLSR, so upstream masters must be configured to keep enough WAL. See [[#Configuration|Configuration]].

==== TRUNCATE is not replicated ====

TRUNCATE is not yet supported, however workarounds with user-level triggers are possible and a ProcessUtility hook is planned to implement a similar approach globally.

The safest option is to define a user-level BEFORE trigger on each table that RAISEs an ERROR when TRUNCATE is attempted.

A simple truncate-blocking trigger is:

CREATE OR REPLACE FUNCTION deny_truncate() RETURNS trigger AS $$
BEGIN
IF tg_op = 'TRUNCATE' THEN
RAISE EXCEPTION 'TRUNCATE is not supported on this table. Please use DELETE FROM.';
ELSE
RAISE EXCEPTION 'This trigger only supports TRUNCATE';
END IF;
END;
$$ LANGUAGE plpgsql;

It can be applied to a table with:

CREATE TRIGGER deny_truncate_on_<tablename> BEFORE TRUNCATE ON <tablename>
FOR EACH STATEMENT EXECUTE PROCEDURE deny_truncate();

A PL/PgSQL DO block that queries <tt>pg_class</tt> and loops over it to <tt>EXECUTE</tt> a dynamic SQL <tt>CREATE TRIGGER</tt> command for each table that does not already have the trigger can be used to apply the trigger to all tables.

=== Initial setup ===

To set up LLSR or BDR you first need a patched PostgreSQL that can support LLSR/BDR, then you need to create one or more LLSR/BDR senders and one or more LLSR/BDR receivers.

==== Installing the patched PostgreSQL binaries ====

Currently BDR is only available in builds of the 'bdr' branch on Andres Freund's git repo on git.postgresql.org. PostgreSQL 9.2 and below do not support BDR, and 9.3 requires patches, so this guide will not work for you if you are trying to use a normal install of PostgreSQL.

First you need to clone, configure, compile and install like normal. Clone the sources from <tt>git://git.postgresql.org/git/users/andresfreund/postgres.git</tt> and checkout the <tt>bdr</tt> branch.

If you have an existing local PostgreSQL git tree specify it as <tt>--reference /path/to/existing/tree</tt> to greatly speed your git clone.

Example:

mkdir -p $HOME/bdr
cd bdr
git clone git://git.postgresql.org/git/users/andresfreund/postgres.git $HOME/bdr/postgres-bdr-src
cd postgres-bdr-src
./configure --prefix=$HOME/bdr/postgres-bdr-bin
make install
cd contrib/bdr
make install

This will put everything in <tt>$HOME/bdr</tt>, with the source code and build tree in <tt>$HOME/bdr/postgres-bdr-src</tt> and the installed PostgreSQL in <tt>$HOME/bdr/postgres-bdr-bin</tt>. This is a convenient setup for testing and development because it doesn't require you to set up new users, wrangle permissions, run anything as root, etc, but it isn't recommended that you deploy this way in production.

To actually use these new binaries you will need to:

export PATH=$HOME/bdr/postgres-bdr-bin/bin:$PATH

before running <tt>initdb</tt>, <tt>postgres</tt>, etc. You don't have to use the <tt>psql</tt> or <tt>libpq</tt> you compiled but you're likely to get version mismatch warnings if you don't.

=== Parameter Reference ===

The following parameters are new or have been changed in PostgreSQL's new logical streaming replication.

==== <tt>shared_preload_libraries = ‘bdr’</tt> ====

To load support for receiving changes on a downstream master, the <tt>bdr</tt> library must be added to the existing ‘shared_preload_libraries’ parameter. This loads the bdr library during postmaster start-up and allows it to create the required background worker(s).

Upstream masters don't need to load the bdr library unless they're also operating as a downstream master as is the case in a BDR configuration.

==== <tt>bdr.connections</tt> ====

A comma-separated list of upstream master connection names is specified in <tt>bdr.connections</tt>. These names must be simple alphanumeric strings. They are used when naming the connection in error messages, configuration options and logs, but are otherwise of no special meaning.

A typical two-upstream-master setting might be:

bdr.connections = ‘upstream1, upstream2’

==== <tt>bdr.<connection_name>.dsn</tt> ====

Each connection name must have at least a data source name specified using the <tt>bdr.<connection_name>.dsn</tt> parameter. The DSN syntax is the same as that used by libpq so it is not discussed in further detail here. A <tt>dbname</tt> for the database to connect to must be specified; all other parts of the DSN are optional.

The local (downstream) database name is assumed to be the same as the name of the upstream database being connected to, though future versions will make this configurable.

For the above two-master setting for <tt>bdr.connections</tt> the DSNs might look like:

bdr.upstream1.dsn = 'host=10.1.1.2 user=postgres dbname=replicated_db'
bdr.upstream2.dsn = 'host=10.1.1.3 user=postgres dbname=replicated_db'

==== <tt>max_logical_slots</tt> ====

The new parameter <tt>max_logical_slots</tt> has been added for use on both upstream and downstream masters. This parameter controls the maximum number of logical replication slots - upstream or downstream - that this cluster may have at a time. It must be set at postmaster start time.

As logical replication slots are persistent, slots are consumed even by replicas that are not currently connected. Slot management is discussed in Starting, Stopping and Managing Replication.

<tt>max_logical_slots</tt> should be set to the sum of the number of logical replication upstream masters this server will have plus the number of logical replication downstream masters will connect to it it.

==== <tt>wal_level = 'logical'</tt> ====

A new setting, <tt>'logical'</tt>, has been added for the existing <tt>wal_level</tt> parameter. <tt>‘logical’</tt> includes everything that the existing <tt>hot_standby</tt> setting does and adds additional details required for logical changeset decoding to the write-ahead logs.

This additional information is consumed by the upstream-master-side xlog decoding worker. Downstream masters that do not also act as upstream masters do not require <tt>wal_level</tt> to be increased above the default <tt>'minimal'</tt>.

<tt>wal_level</tt>, except for the new <tt>'logical'</tt> setting, is [http://www.postgresql.org/docs/current/static/runtime-config-wal.html documented in the main PostgreSQL manual].

==== <tt>max_wal_senders</tt> ====

Logical replication hasn't altered the <tt>max_wal_senders</tt> parameter, but it is important in upstream masters for logical replication and BDR because every logical sender consumes a <tt>max_wal_senders</tt> entry.

You should configure <tt>max_wal_senders</tt> to the sum of the number of physical and logical replicas you want to allow an upstream master to serve. If you intend to use <tt>pg_basebackup</tt> you should add at least two more senders to allow for its use.

Like <tt>max_logical_slots</tt>, <tt>max_wal_senders</tt> entries don't cost a large amount of memory, so you can overestimate fairly safely.

<tt>max_wal_senders</tt> is documented in [http://www.postgresql.org/docs/current/static/runtime-config-replication.html the main PostgreSQL documentation].

==== <tt>wal_keep_segments</tt> ====

Like <tt>max_wal_senders</tt>, the <tt>wal_keep_segments</tt> parameter isn't directly changed by logical replication but is still important for upstream masters. It is not required on downstream-only masters.

<tt>wal_keep_segments</tt> should be set to a value that allows for some downtime or unreachable periods for downstream masters and for heavy bursts of write activity on the upstream master.

Keep in mind that enough disk space must be available for the WAL segments, each of which is 16MB. If you run out of disk space the server will halt until disk space is freed and it may be quite difficult to free space when you can no longer start the server.

If you exceed the required <tt>wal_keep_segments</tt> and "Insufficient WAL segments retained" error will be reported. See [[#Troubleshooting|Troubleshooting]].

<tt>wal_keep_segments</tt> is documented in the [http://www.postgresql.org/docs/current/static/runtime-config-replication.html the main PostgreSQL manual].

==== <tt>track_commit_timestamp</tt> ====

Setting this parameter to "on" enables commit timestamp tracking, which is used to implement last-UPDATE-wins conflict resolution.

=== Configuration ===

Details on individual parameters are described in the [[parameter reference]] section.

The following configuration is an example of a simple one-way LLSR replication setup - a single upstream master to a single downstream master.

The upstream master (sender)'s <tt>postgresql.conf</tt> should contain settings like:

wal_level = 'logical' # Include enough info for logical replication
max_logical_slots = X # Number of LLSR senders + any receivers
max_wal_senders = Y # Y = max_logical_slots plus any physical
# streaming requirements
wal_keep_segments = 5000 # Master must retain enough WAL segments to let
# replicas catch up. Correct value depends on
# rate of writes on master, max replica downtime
# allowable. 5000 segments requires 78GB
# in pg_xlog

Downstream (receiver) <tt>postgresql.conf</tt>:

shared_preload_libraries = 'bdr'

bdr.connections="name_of_upstream_master" # list of upstream master nodenames
bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection
# from downstream to upstream master
bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case
# where the databasename on upstream
# and downstream master differ.
# (Not yet implemented)
bdr.<nodename>.apply_delay # optional parameter to delay apply of
# transactions, time in milliseconds
bdr.synchronous_commit = ...; # optional parameter to set the
# synchronous_commit parameter the
# apply processes will be using
max_logical_slots = X # set to the number of remotes

Note that a server can be both sender and receiver, either two servers to each other or more complex configurations like replication chains/trees.

The upstream (sender) <tt>pg_hba.conf</tt> must be configured to allow the downstream master to connect for replication. Otherwise you'll see errors like the following on the downstream master:

FATAL: could not connect to the primary server: FATAL: no pg_hba.conf entry for replication connection from host "[local]", user "postgres"

A suitable <tt>pg_hba.conf</tt> entry for a replication connection from the replica server 10.1.4.8 might be:

host replication postgres 10.1.4.8/32 trust

(the user name should match the user name configured in the downstream master's dsn. md5 password authentication is supported.)

For more details on these parameters, see [[#Parameter Reference|Parameter Reference]].

=== Troubleshooting ===

==== Could not access file "bdr": No such file or directory ====

If you see the error:

FATAL: could not access file "bdr": No such file or directory

when starting a database set up to receive BDR replication, you probably forgot to install <tt>contrib/bdr</tt>. See above.

==== Invalid value for parameter ====

An error like:

LOG: invalid value for parameter ...

when setting one of these parameters means your server doesn't support logical replication and will need to be patched or updated.

==== Insufficient WAL segments retained ("requested WAL segment ... has already been removed") ====

If <tt>wal_keep_segments</tt> is insufficient to meet the requirements of a replica that has fallen far behind, the master will report errors like:

ERROR: requested WAL segment 00000001000000010000002D has already been removed

Currently the replica errors look like:

WARNING: Starting logical replication
LOG: data stream ended
LOG: worker process: master (PID 23812) exited with exit code 0
LOG: starting background worker process "master"
LOG: master initialized on master, remote dbname=master port=5434 replication=true fallback_application_name=bdr
LOG: local sysid 5873181566046043070, remote: 5873181102189050714
LOG: found valid replication identifier 1
LOG: starting up replication at 1 from 1/2D9CA220

but a more explicit error message for this condition is planned.

The only way to recover from this fault is to re-seed the replica database.

This fault could be prevented with feedback from the replica to the master, but this feature is not planned for the first release of BDR. Another alternative considered for future releases is making wal_keep_segments a dynamic parameter that is sized on demand.

Monitoring of maximum replica lag and appropriate adjustment of wal_keep_segments will prevent this fault from arising.

==== Couldn't find logical slot ====

An error like:

ERROR: couldn't find logical slot "bdr: 16384:5873181566046043070-1-24596:"

on the upstream master suggests that a downstream master is trying to connect to a logical replication slot that no longer exists. The slot can not be re-created, so it is necessary to re-seed the downstream replica database.

=== Operational Issues and Debugging ===

In LLSR there are no user-level (ie SQL visible) ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

The following views are available for monitoring replication activity:

* <tt>[http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE pg_stat_replication]</tt>
* <tt>pg_stat_logical_replication</tt> (described below)
* <tt>pg_stat_bdr</tt> (described below)

The following configuration and logging parameters are useful for monitoring replication:

* <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt>

==== pg_stat_logical_replication ====

The new <tt>pg_stat_logical_replication</tt> view is specific to logical replication. It is based on the underlying <tt>pg_stat_get_logical_replication_slots</tt> function and has the following structure:

View "pg_catalog.pg_stat_logical_replication"
Column | Type | Modifiers
--------------------------+---------+-----------
slot_name | text |
plugin | text |
database | oid |
active | boolean |
xmin | xid |
last_required_checkpoint | text |

It contains one row for every connection from a downstream master to the server being queried (the upstream master). On a standalone PostgreSQL server or a downstream-only master this view will contain no rows.

* <tt>slot_name</tt>: An internal name for a given logical replication slot (a connection from a downstream master to this upstream master). This slot name is used by the downstream master to uniquely identify its self and is used with the <tt>pg_receivellog</tt> command when managing logical replication slots. The slot name is composed of the decoding plugin name, the upstream database oid, the downstream system identifier (from <tt>pg_control</tt>), the downstream slot number, and the downstream database oid.

* <tt>plugin</tt>: The logical replication plugin being used to decode WAL archives. You'll generally only see <tt>bdr_output</tt> here.

* <tt>database</tt>: The oid of the database being replicated by this slot. You can get the database name by joining on <tt>pg_database.oid</tt>.

* <tt>active</tt>: Whether this slot currently has an active connection.

* <tt>xmin</tt>: The lowest transaction ID this replication slot can "see", like the xmin of a transaction or prepared transaction. xmin should keep on advancing as replication continues.

* <tt>last_required_checkpoint</tt>: The checkpoint identifying the oldest WAL record required to bring this slot up to date with the upstream master. (This column is likely to be removed in a future version).

==== pg_stat_bdr ====

The <tt>pg_stat_bdr</tt> view is supplied by the <tt>bdr</tt> extension. It provides information on a server's connection(s) to its upstream master(s). It is not present on upstream-only masters.

The primary purpose of this view is to report statistics on the progress of LLSR apply on a per-upstream master connection basis.

View structure:

View "public.pg_stat_bdr"
Column | Type | Modifiers
--------------------+--------+-----------
rep_node_id | oid |
riremotesysid | name |
riremotedb | oid |
rilocaldb | oid |
nr_commit | bigint |
nr_rollback | bigint |
nr_insert | bigint |
nr_insert_conflict | bigint |
nr_update | bigint |
nr_update_conflict | bigint |
nr_delete | bigint |
nr_delete_conflict | bigint |
nr_disconnect | bigint |

Fields:

* <tt>rep_node_id</tt>: An internal identifier for the replication slot.

* <tt>riremotesysid</tt>: The remote database system identifier, as reported by the <tt>Database system identifier</tt> line of <tt>pg_controldata /path/to/datadir</tt>

* <tt>riremotedb</tt>: The remote database OID, ie the <tt>oid</tt> column of the remote server's <tt>pg_catalog.pg_database</tt> entry for the replicated database. You can get the database name with <tt>select datname from pg_database where oid = 12345</tt> (where '12345' is the <tt>riremotedb</tt> oid).

* <tt>rilocaldb </tt>: The local database OID, with the same meaning as <tt>riremotedb</tt> but with oids from the local system.

''The rest of the rows are statistics about this upstream master slot'':

* <tt>nr_commit</tt>: Number of commits applied to date from this master

* <tt>nr_rollback</tt>: Number of rollbacks performed by this apply process due to recoverable errors (deadlock retries, lost races, etc) or unrecoverable errors like mismatched constraint errors.

* <tt>nr_insert</tt>: Number of <tt>INSERT</tt>s performed

* <tt>nr_insert_conflict</tt>: Number of <tt>INSERT</tt>s that resulted in conflicts.

* <tt>nr_update</tt>: Number of <tt>UPDATE</tt>s performed

* <tt>nr_update_conflict</tt>: Number of <tt>UPDATE</tt>s that resulted in conflicts.

* <tt>nr_delete</tt>: Number of deletes performed

* <tt>nr_delete_conflict</tt>: Number of deletes that resulted in conflicts.

* <tt>nr_disconnect</tt>: Number of times this apply process has lost its connection to the upstream master since it was started.

This view does not contain any information about how far behind the upstream master this downstream master is. The upstream master's <tt>pg_stat_logical_replication</tt> and <tt>pg_stat_replication</tt> views must be queried to determine replication lag.

==== Monitoring replication status and lag ====

As with any replication setup, it is vital to monitor replication status on all BDR nodes to ensure no node is lagging severely behind the others or is stuck.

In the case of BDR a stuck or crashed node will eventually cause disk space and table bloat problems on other masters so stuck nodes should be detected and removed or repaired in a reasonably timely manner. Exactly how urgent this is depends on the workload of the BDR group.

The <tt>pg_stat_logical_replication</tt> view described above may be used to verify that a downstream master is connected to its upstream master - the <tt>active</tt> boolean column is <tt>t</tt> if there's a downstream master connected.

The <tt>xmin</tt> column provides an indication of whether replication is advancing; it should increase as replication progresses. There is no simple way to turn <tt>xmin</tt> into the time the last applied transaction was committed on the master, so it doesn't provide an indication of wall-clock lag.

To determine wall-clock replication lag an application-level ticker may be used to periodically update a timestamp in a replicated table. The difference between this timestamp on the upstream and downstream masters provides the wall-clock replication lag. For BDR one row may be added to the table for each BDR master, giving an indication of how much lag each master has relative to each other master.

=== Table and index usage statistics ===

Statistics on table and index usage are updated normally by the downstream master. This is essential for correct function of auto-vacuum. If there are no local writes on the downstream master and stats have not been reset these two views should show matching results between upstream and downstream:

* <tt>pg_stat_user_tables</tt>
* <tt>pg_statio_user_tables</tt>

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform <tt>UPDATE</tt>s and <tt>DELETE</tt>s than non-identifying indexes are.

The built-in index monitoring views are:

* <tt>pg_stat_user_indexes</tt>
* <tt>pg_statio_user_indexes</tt>

All these views are discussed in [http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE the PostgreSQL documentation on the statistics views].

=== Starting, stopping and managing replication ===

Replication is managed with the <tt>postgresql.conf</tt> settings described in "Parameter Reference" and "Configuration" above, and using the <tt>pg_receivellog</tt> utility command.

==== Starting a new LLSR connection ====

Logical replication is started automatically when a database is configured as a downstream master in <tt>postgresql.conf</tt> (see [[#Configuration|Configuration]]) and the postmaster is started. No explicit action is required to start replication, but replication will not actually work unless the upstream and downstream databases are identical within the requirements set by LLSR in the [[#Table definitions and DDL replication||Table definitions and DDL replication]] section.

<tt>pg_dump</tt> and <tt>pg_restore</tt> may be used to set up the new replica's database.

==== Viewing logical replication slots ====

Examining the state of logical replication is discussed in [[#Monitoring|Monitoring]].

==== Temporarily stopping an LLSR replica ====

LLSR replicas can be temporarily stopped by shutting down the downstream master's postmaster.

If the replica is not started back up before the upstream master discards the oldest WAL segment required for the downstream master to resume replay, as identified by the <tt>last_required_checkpoint</tt> column of <tt>pg_catalog.pg_stat_logical_replication</tt> then the replica will not resume replay. The error [[#Insufficient_WAL_segments_retained_.28.22requested_WAL_segment_..._has_already_been_removed.22.29|Insufficient WAL segments retained]] will be reported in the upstream master's logs. The replica must be re-created for replication to continue.

==== Removing an LLSR replica permanently ====

To remove a replication connection permanently, remove its entries from the downstream master's <tt>postgresql.conf</tt>, restart the downstream master, then use <tt>pg_receivellog</tt> to remove the replication slot on the upstream master.

It is important to remove the replication slot from the upstream master(s) to prevent xid wrap-around problems and issues with table bloat caused by delayed vacuum.

==== Cleaning up abandoned replication slots ====

To remove a replication slot that was used for a now-defunct replica, find its slot name from the <tt>[[#pg_stat_logical_replication|pg_stat_logical_replication]]</tt> view on the upstream master then run:

pg_receivellog -p 5434 -h master-hostname -d dbname \
--slot='bdr: 16384:5873181566046043070-1-16384:' --stop

where the argument to '--slot' is the slot name you found from the view.

You may need to do this if you've created and then deleted several replicas so <tt>max_logical_slots</tt> has filled up with entries for replicas that no longer exist.

== Bi-Directional Replication ==

Bi-Directional replication is built directly on LLSR by configuring two or more servers as both upstream ''and'' downstream masters of each other.

All of the Log Level Streaming Replication documentation applies to BDR and should be read before moving on to reading about and configuring BDR.

=== Bi-Directional Replication Use Cases ===

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

==== Simple multi-master pair ====

A simple mulit-master "HA Cluster" with two servers:

* Server "Alpha" - Master
* Server "Bravo" - Master

===== Configuration =====

Alpha:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="bravo"
bdr.bravo.dsn = 'dbname=dbtoreplicate'
track_commit_timestamp = on

Bravo:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="alpha"
bdr.alpha.dsn = 'dbname=dbtoreplicate'
track_commit_timestamp = on

See [[#Configuration|Configuration]] for an explanation of these parameters.

==== HA and Logical Standby ====
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

"HA Cluster":

* Server "Alpha" - Current Master
* Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
* Server "Charlie" - "Logical Standby" - downstream master

==== Very High Availability Multi-Master ====
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

==== 3-remote site simple Multi-Master Plex ====

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Alpha, Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Charlie using logical streaming replication

===== Configuration =====

If you wanted to test this configuration locally you could run three PostgreSQL instances on different ports. Such a configuration would look like the following if the port numbers were used as node names for the sake of notational clarity:

Config for node_5440:

port = 5440
bdr.connections='node_5441,node_5442'
bdr.node_5441.dsn='port=5441 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5440,node_5442'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440,node_5441'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5441.dsn='port=5441 dbname=postgres'

In a typical real-world configuration each server would be on the same port on a different host instead.

==== 3-remote site simple Multi-Master Circular Replication ====

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases. It's also less resilient to network disruptions and node faults.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

TODO: Regrettably this doesn't actually work yet because we don't cascade logical changes (yet).

===== Configuration =====

Using node names that match port numbers, for clarity

Config for node_5440:

port = 5440
bdr.connections='node_5441'
bdr.node_5441.dsn='port=5441 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5442'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440'
bdr.node_5440.dsn='port=5440 dbname=postgres'

This would usually be done in the real world with databases on different hosts, all running on the same port.

==== 3-remote site Max Availability Multi-Master Plex ====

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

Config left as an exercise for the reader.

==== N-site symmetric cluster replication ====

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

==== Complex/Assymetric Replication ====

Variety of options are possible.

=== Conflict Avoidance ===

==== Distributed Locking ====

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications as very low latency is critical for acceptable performance.

Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible but allow some types of conflict to occur and and resolve them when they arise.

==== Global Sequences ====

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

The SQL standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using <tt>DEFAULT nextval('mysequence')</tt>, as with PostgreSQL's <tt>SERIAL</tt> pseudo-type.

BDR requires sequences to work together across multiple nodes. This is implemented as a new <tt>SequenceAccessMethod</tt> API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

=== Conflict Detection & Resolution ===

Because local writes can occur on a master, conflict detection and avoidance is a concern for basic LLSR setups as well as full BDR configurations.

==== Lock Conflicts ====

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt> facility to look for issues with apply blocking on locks.

By concurrent activity on a row, we include

* explicit row level locking (<tt>SELECT ... FOR UPDATE/FOR SHARE</tt>)
* locking from foreign keys
* implicit locking because of row <tt>UPDATE</tt>s, <tt>INSERT</tt>s or <tt>DELETE</tt>s, either from local activity or apply from other servers

==== Data Conflicts ====

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in a consistent and idempotent manner so that all servers end up with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from <tt>pg_control</tt> though this may change in a future release.

<tt>UPDATE</tt>s and <tt>INSERT</tt>s may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur. The errors causing the conflict can be seen in the error log of the downstream master with the problem.

Updates which cannot locate a row are presumed to be <tt>DELETE</tt>/<tt>UPDATE</tt> conflicts. These are accepted as successful operations but in the case of <tt>UPDATE</tt> the data in the <tt>UPDATE</tt> is discarded.

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins. It is not practical to decide when a row should be merged and when a last-update-wins stragegy should be used at the database level; such decision making would require support for application-specific conflict resolution plugins.

Changing unlogged and logged tables in the same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

==== Examples ====

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-05-16T13:30:50Z

Simon: /* Replication of DML changes */

----
This page is the users and administrators guide for BDR. If you're looking for technical details on the project plan and implementation, see [[BDR Project]].
----

= BDR User Guide =

BDR (BiDrectional Replication) is a feature being developed for inclusion in PostgreSQL core that provides greatly enhanced replication capabilities.

BDR allows users to create a geographically distributed multi-master database using Logical Log Streaming Replication (LLSR) transport.
BDR is designed to provide both high availability and geographically distributed disaster recovery capabilities.

BDR is not “clustering” as some vendors use the term, in that it doesn't have a distributed lock manager, global transaction co-ordinator, etc. Each member server is separate yet connected, with design choices that allow separation between nodes that would not be possible with global transaction coordination.

Guidance on getting a testing setup established are in [[#Initial setup]]. Please read the full documentation if you intend to put BDR into production.

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows one PostgreSQL master (the "upstream master") to stream a sequence of changes to another read/write PostgreSQL server (the "downstream master"). Data is sent in one direction only over a normal libpq connection.

Multiple LLSR connections can be used to set up bi-directional replication as discussed later in this guide.

=== Overview of logical replication ===

In some ways LLSR is similar to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective; both replicate changes from one server to another. However, in LLSR the receiving server is also a full master database that can make changes, unlike the read-only replicas offered by PLSR hot standby. Additionally, LLSR is per-database, whereas PLSR is per-cluster and replicates all databases at once. There are many more differences discussed in the relevant sections of this document.

In LLSR the data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after <tt>CREATE DATABASE</tt>. A restart of the downstream master is also required. The upstream master only needs restarting if the <tt>max_logical_slots</tt> parameter is too low to allow a new replica to be added. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated. Setup is discussed in more detail below.

Changes are processed by the downstream master using <tt>bdr</tt> plug-ins. This allows flexible handing of replication input, including:

* BDR apply process - applies logical changes to the downstream master. The apply process makes changes directly rather than generating SQL text and then parse/plan/executing SQL.
* Textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* <tt>pg_xlogdump</tt> - examines physical WAL records and produces textual debugging output. This server program is included in PostgreSQL 9.3.

=== Replication of DML changes ===

All changes are replicated: <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> and <tt>TRUNCATE</tt>.

(TRUNCATE is not yet implemented, but will be implemented before the feature goes to final release).

Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though it has overheads that mean that it doesn't always use less bandwidth than PLSR.

Locks taken by <tt>LOCK</tt> and <tt>SELECT ... FOR UPDATE/SHARE</tt> on the upstream master are not replicated to downstream masters. Locks taken automatically by <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> or <tt>TRUNCATE</tt> *are* taken on the downstream master and may delay replication apply or concurrent transactions - see [[#Lock Conflicts|Lock Conflicts]].

<tt>TEMPORARY</tt> and <tt>UNLOGGED</tt> tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables. However, temporary tables remain specific to a particular session so creating a temporary table on the upstream master does not create a similar table on the downstream master.

<tt>DELETE</tt> and <tt>UPDATE</tt> statements that affect multiple rows on upstream master will cause a series of row changes on downstream master. These are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. <tt>INSERT</tt> on upstream master do not require a unique constraint in order to replicate correctly. <tt>UPDATE</tt>s and <tt>DELETE</tt>s require some form of unique constraint, either <tt>PRIMARY KEY</tt> or <tt>UNIQUE NOT NULL</tt>. A warning is issued in the downstream master's logs if the expected constraint is absent.

<tt>UPDATE</tt>s that change the value of the Primary Key of a table will be replicated correctly.

The values applied are the final values from the <tt>UPDATE</tt> on the upstream master, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value. Volatile or stable functions are evaluated on the master side and the resulting values are replicated. Consequently any function side-effects (writing files, network socket activity, updating internal PostgreSQL variables, etc) will not occur on the replicas as the functions are not run again on the replica.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master (see "Limitations").

The current LLSR plugin implementation uses the binary libpq protocol, so it requires that the upstream and downstream master use same CPU architecture and word-length, i.e. "identical servers", as with physical replication. A textual output option will be added later for passing data between non-identical servers, e.g. laptops or mobile devices communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is efficiently implemented. Parallel apply is a possible future feature, especially for changes made while holding <tt>AccessExclusiveLock</tt>.

Changes are applied to the downstream master in the sequence in which they were commited on the upstream master. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions spill to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

<tt>SET</tt> statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. We always update the correct tables, whatever the setting of <tt>search_path</tt>. Values are replicated correctly irrespective of the values of <tt>bytea_output</tt>, <tt>TimeZone</tt>, <tt>DateStyle</tt>, etc.

<tt>NOTIFY</tt> is not supported across log based replication, either physical or logical. <tt>NOTIFY</tt> and <tt>LISTEN</tt> will work fine on the upstream master but an upstream <tt>NOTIFY</tt> will not trigger a downstream <tt>LISTEN</tt>er.

In some cases, additional deadlocks can occur on apply. This causes an automatic retry of the apply of the replaying transaction and is only an issue if the deadlock recurs repeatedly, delaying replication.

From a performance and concurrency perspective the BDR apply process is similar to a normal backend. Frequent conflicts with locks from other transactions when replaying changes can slow things down and thus increase replication delay, so reducing the frequency of such conflicts can be a good way to speed things up. Any lock held by another transaction on the downstream master - <tt>LOCK</tt> statements, <tt>SELECT ... FOR UPDATE/FOR SHARE</tt>, or <tt>INSERT</tt>/<tt>UPDATE</tt>/<tt>DELETE</tt> row locks - can delay replication if the replication apply process needs to change the locked table/row.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching <tt>"Schemaname"."Tablename"</tt> on both upstream and downstream masters. e.g. changes from upstream's <tt>public.mytable</tt> will go to downstream's <tt>public.mytable</tt> while changes to the upstream <tt>mychema.mytable</tt> will go to the downstream <tt>myschema.mytable</tt>. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful synchronization of table definitions on each node otherwise <tt>ERROR</tt>s will be generated by the replication apply process. In general, tables must be an exact match between upstream and downstream masters.

There are no plans to implement working replication between dissimilar table definitions.

Tables must meet the following requirements to be compatible for purposes of LLSR:

* The downstream master must only have constraints (<tt>CHECK</tt>, <tt>UNIQUE</tt>, <tt>EXCLUSION</tt>, <tt>FOREIGN KEY</tt>, etc) that are also present on the upstream master. Replication may initially work with mismatched constraints but is likely to fail as soon as the downstream master rejects a row the upstream master accepted.
* The table referenced by a FOREIGN KEY on a downstream master must have all the keys present in the upstream master version of the same table.
* Storage parameters must match except for as allowed below
* Inheritance must be the same
* Dropped columns on master must be present on replicas
* Custom types and enum definitions must match exactly
* Composite types and enums must have the same oids on master and replication target
* Extensions defining types used in replicated tables must be of the same version or fully SQL-level compatible and the oids of the types they define must match.

The following differences are permissible between tables on different nodes:

* The table's <tt>pg_class</tt> oid, the oid of its associated TOAST table, and the oid of the table's rowtype in <tt>pg_type</tt> may differ;
* Extra or missing non-<tt>UNIQUE</tt> indexes
* Extra keys in downstream lookup tables for <tt>FOREIGN KEY</tt> references that are not present on the upstream master
* The table-level storage parameters for fillfactor and autovacuum
* Triggers and rules may differ (they are not executed by replication apply)

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR (see [[#LLSR Limitations|LLSR Limitations]]).

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of <tt>session_replication_role = origin</tt>.

In future it is expected that composite types and enums with non-identical oids will be converted using text output and input functions. This feature is not yet implemented.

=== LLSR limitations ===

The current LLSR implementation is subject to some limitations, which are being progressively removed as work progresses.

==== Data definition compatibility ====

Table definitions, types, extensions, etc must be near identical between upstream and downstream masters. See [[#Table definitions and DDL replication|Table definitions and DDL replication]].

==== DDL Replication ====

DDL replication is not yet supported.

==== Upstream feedback ====

No feedback from downstream masters to the upstream master is implemented for asynchronous LLSR, so upstream masters must be configured to keep enough WAL. See [[#Configuration|Configuration]].

==== TRUNCATE is not replicated ====

TRUNCATE is not yet supported, however workarounds with user-level triggers are possible and a ProcessUtility hook is planned to implement a similar approach globally.

The safest option is to define a user-level BEFORE trigger on each table that RAISEs an ERROR when TRUNCATE is attempted.

A simple truncate-blocking trigger is:

CREATE OR REPLACE FUNCTION deny_truncate() RETURNS trigger AS $$
BEGIN
IF tg_op = 'TRUNCATE' THEN
RAISE EXCEPTION 'TRUNCATE is not supported on this table. Please use DELETE FROM.';
ELSE
RAISE EXCEPTION 'This trigger only supports TRUNCATE';
END IF;
END;
$$ LANGUAGE plpgsql;

It can be applied to a table with:

CREATE TRIGGER deny_truncate_on_<tablename> BEFORE TRUNCATE ON <tablename>
FOR EACH STATEMENT EXECUTE PROCEDURE deny_truncate();

A PL/PgSQL DO block that queries <tt>pg_class</tt> and loops over it to <tt>EXECUTE</tt> a dynamic SQL <tt>CREATE TRIGGER</tt> command for each table that does not already have the trigger can be used to apply the trigger to all tables.

=== Initial setup ===

To set up LLSR or BDR you first need a patched PostgreSQL that can support LLSR/BDR, then you need to create one or more LLSR/BDR senders and one or more LLSR/BDR receivers.

==== Installing the patched PostgreSQL binaries ====

Currently BDR is only available in builds of the 'bdr' branch on Andres Freund's git repo on git.postgresql.org. PostgreSQL 9.2 and below do not support BDR, and 9.3 requires patches, so this guide will not work for you if you are trying to use a normal install of PostgreSQL.

First you need to clone, configure, compile and install like normal. Clone the sources from <tt>git://git.postgresql.org/git/users/andresfreund/postgres.git</tt> and checkout the <tt>bdr</tt> branch.

If you have an existing local PostgreSQL git tree specify it as <tt>--reference /path/to/existing/tree</tt> to greatly speed your git clone.

Example:

mkdir -p $HOME/bdr
cd bdr
git clone git://git.postgresql.org/git/users/andresfreund/postgres.git $HOME/bdr/postgres-bdr-src
cd postgres-bdr-src
./configure --prefix=$HOME/bdr/postgres-bdr-bin
make install
cd contrib/bdr
make install

This will put everything in <tt>$HOME/bdr</tt>, with the source code and build tree in <tt>$HOME/bdr/postgres-bdr-src</tt> and the installed PostgreSQL in <tt>$HOME/bdr/postgres-bdr-bin</tt>. This is a convenient setup for testing and development because it doesn't require you to set up new users, wrangle permissions, run anything as root, etc, but it isn't recommended that you deploy this way in production.

To actually use these new binaries you will need to:

export PATH=$HOME/bdr/postgres-bdr-bin/bin:$PATH

before running <tt>initdb</tt>, <tt>postgres</tt>, etc. You don't have to use the <tt>psql</tt> or <tt>libpq</tt> you compiled but you're likely to get version mismatch warnings if you don't.

=== Parameter Reference ===

The following parameters are new or have been changed in PostgreSQL's new logical streaming replication.

==== <tt>shared_preload_libraries = ‘bdr’</tt> ====

To load support for receiving changes on a downstream master, the <tt>bdr</tt> library must be added to the existing ‘shared_preload_libraries’ parameter. This loads the bdr library during postmaster start-up and allows it to create the required background worker(s).

Upstream masters don't need to load the bdr library unless they're also operating as a downstream master as is the case in a BDR configuration.

==== <tt>bdr.connections</tt> ====

A comma-separated list of upstream master connection names is specified in <tt>bdr.connections</tt>. These names must be simple alphanumeric strings. They are used when naming the connection in error messages, configuration options and logs, but are otherwise of no special meaning.

A typical two-upstream-master setting might be:

bdr.connections = ‘upstream1, upstream2’

==== <tt>bdr.<connection_name>.dsn</tt> ====

Each connection name must have at least a data source name specified using the <tt>bdr.<connection_name>.dsn</tt> parameter. The DSN syntax is the same as that used by libpq so it is not discussed in further detail here. A <tt>dbname</tt> for the database to connect to must be specified; all other parts of the DSN are optional.

The local (downstream) database name is assumed to be the same as the name of the upstream database being connected to, though future versions will make this configurable.

For the above two-master setting for <tt>bdr.connections</tt> the DSNs might look like:

bdr.upstream1.dsn = 'host=10.1.1.2 user=postgres dbname=replicated_db'
bdr.upstream2.dsn = 'host=10.1.1.3 user=postgres dbname=replicated_db'

==== <tt>max_logical_slots</tt> ====

The new parameter <tt>max_logical_slots</tt> has been added for use on both upstream and downstream masters. This parameter controls the maximum number of logical replication slots - upstream or downstream - that this cluster may have at a time. It must be set at postmaster start time.

As logical replication slots are persistent, slots are consumed even by replicas that are not currently connected. Slot management is discussed in Starting, Stopping and Managing Replication.

<tt>max_logical_slots</tt> should be set to the sum of the number of logical replication upstream masters this server will have plus the number of logical replication downstream masters will connect to it it.

==== <tt>wal_level = 'logical'</tt> ====

A new setting, <tt>'logical'</tt>, has been added for the existing <tt>wal_level</tt> parameter. <tt>‘logical’</tt> includes everything that the existing <tt>hot_standby</tt> setting does and adds additional details required for logical changeset decoding to the write-ahead logs.

This additional information is consumed by the upstream-master-side xlog decoding worker. Downstream masters that do not also act as upstream masters do not require <tt>wal_level</tt> to be increased above the default <tt>'minimal'</tt>.

<tt>wal_level</tt>, except for the new <tt>'logical'</tt> setting, is [http://www.postgresql.org/docs/current/static/runtime-config-wal.html documented in the main PostgreSQL manual].

==== <tt>max_wal_senders</tt> ====

Logical replication hasn't altered the <tt>max_wal_senders</tt> parameter, but it is important in upstream masters for logical replication and BDR because every logical sender consumes a <tt>max_wal_senders</tt> entry.

You should configure <tt>max_wal_senders</tt> to the sum of the number of physical and logical replicas you want to allow an upstream master to serve. If you intend to use <tt>pg_basebackup</tt> you should add at least two more senders to allow for its use.

Like <tt>max_logical_slots</tt>, <tt>max_wal_senders</tt> entries don't cost a large amount of memory, so you can overestimate fairly safely.

<tt>max_wal_senders</tt> is documented in [http://www.postgresql.org/docs/current/static/runtime-config-replication.html the main PostgreSQL documentation].

==== <tt>wal_keep_segments</tt> ====

Like <tt>max_wal_senders</tt>, the <tt>wal_keep_segments</tt> parameter isn't directly changed by logical replication but is still important for upstream masters. It is not required on downstream-only masters.

<tt>wal_keep_segments</tt> should be set to a value that allows for some downtime or unreachable periods for downstream masters and for heavy bursts of write activity on the upstream master.

Keep in mind that enough disk space must be available for the WAL segments, each of which is 16MB. If you run out of disk space the server will halt until disk space is freed and it may be quite difficult to free space when you can no longer start the server.

If you exceed the required <tt>wal_keep_segments</tt> and "Insufficient WAL segments retained" error will be reported. See [[#Troubleshooting|Troubleshooting]].

<tt>wal_keep_segments</tt> is documented in the [http://www.postgresql.org/docs/current/static/runtime-config-replication.html the main PostgreSQL manual].

==== <tt>track_commit_timestamp</tt> ====

Setting this parameter to "on" enables commit timestamp tracking, which is used to implement last-UPDATE-wins conflict resolution.

=== Configuration ===

Details on individual parameters are described in the [[parameter reference]] section.

The following configuration is an example of a simple one-way LLSR replication setup - a single upstream master to a single downstream master.

The upstream master (sender)'s <tt>postgresql.conf</tt> should contain settings like:

wal_level = 'logical' # Include enough info for logical replication
max_logical_slots = X # Number of LLSR senders + any receivers
max_wal_senders = Y # Y = max_logical_slots plus any physical
# streaming requirements
wal_keep_segments = 5000 # Master must retain enough WAL segments to let
# replicas catch up. Correct value depends on
# rate of writes on master, max replica downtime
# allowable. 5000 segments requires 78GB
# in pg_xlog

Downstream (receiver) <tt>postgresql.conf</tt>:

shared_preload_libraries = 'bdr'

bdr.connections="name_of_upstream_master" # list of upstream master nodenames
bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection
# from downstream to upstream master
bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case
# where the databasename on upstream
# and downstream master differ.
# (Not yet implemented)
bdr.<nodename>.apply_delay # optional parameter to delay apply of
# transactions, time in milliseconds
bdr.synchronous_commit = ...; # optional parameter to set the
# synchronous_commit parameter the
# apply processes will be using
max_logical_slots = X # set to the number of remotes

Note that a server can be both sender and receiver, either two servers to each other or more complex configurations like replication chains/trees.

The upstream (sender) <tt>pg_hba.conf</tt> must be configured to allow the downstream master to connect for replication. Otherwise you'll see errors like the following on the downstream master:

FATAL: could not connect to the primary server: FATAL: no pg_hba.conf entry for replication connection from host "[local]", user "postgres"

A suitable <tt>pg_hba.conf</tt> entry for a replication connection from the replica server 10.1.4.8 might be:

host replication postgres 10.1.4.8/32 trust

(the user name should match the user name configured in the downstream master's dsn. md5 password authentication is supported.)

For more details on these parameters, see [[#Parameter Reference|Parameter Reference]].

=== Troubleshooting ===

==== Could not access file "bdr": No such file or directory ====

If you see the error:

FATAL: could not access file "bdr": No such file or directory

when starting a database set up to receive BDR replication, you probably forgot to install <tt>contrib/bdr</tt>. See above.

==== Invalid value for parameter ====

An error like:

LOG: invalid value for parameter ...

when setting one of these parameters means your server doesn't support logical replication and will need to be patched or updated.

==== Insufficient WAL segments retained ("requested WAL segment ... has already been removed") ====

If <tt>wal_keep_segments</tt> is insufficient to meet the requirements of a replica that has fallen far behind, the master will report errors like:

ERROR: requested WAL segment 00000001000000010000002D has already been removed

Currently the replica errors look like:

WARNING: Starting logical replication
LOG: data stream ended
LOG: worker process: master (PID 23812) exited with exit code 0
LOG: starting background worker process "master"
LOG: master initialized on master, remote dbname=master port=5434 replication=true fallback_application_name=bdr
LOG: local sysid 5873181566046043070, remote: 5873181102189050714
LOG: found valid replication identifier 1
LOG: starting up replication at 1 from 1/2D9CA220

but a more explicit error message for this condition is planned.

The only way to recover from this fault is to re-seed the replica database.

This fault could be prevented with feedback from the replica to the master, but this feature is not planned for the first release of BDR. Another alternative considered for future releases is making wal_keep_segments a dynamic parameter that is sized on demand.

Monitoring of maximum replica lag and appropriate adjustment of wal_keep_segments will prevent this fault from arising.

==== Couldn't find logical slot ====

An error like:

ERROR: couldn't find logical slot "bdr: 16384:5873181566046043070-1-24596:"

on the upstream master suggests that a downstream master is trying to connect to a logical replication slot that no longer exists. The slot can not be re-created, so it is necessary to re-seed the downstream replica database.

=== Operational Issues and Debugging ===

In LLSR there are no user-level (ie SQL visible) ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

The following views are available for monitoring replication activity:

* <tt>[http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE pg_stat_replication]</tt>
* <tt>pg_stat_logical_replication</tt> (described below)
* <tt>pg_stat_bdr</tt> (described below)

The following configuration and logging parameters are useful for monitoring replication:

* <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt>

==== pg_stat_logical_replication ====

The new <tt>pg_stat_logical_replication</tt> view is specific to logical replication. It is based on the underlying <tt>pg_stat_get_logical_replication_slots</tt> function and has the following structure:

View "pg_catalog.pg_stat_logical_replication"
Column | Type | Modifiers
--------------------------+---------+-----------
slot_name | text |
plugin | text |
database | oid |
active | boolean |
xmin | xid |
last_required_checkpoint | text |

It contains one row for every connection from a downstream master to the server being queried (the upstream master). On a standalone PostgreSQL server or a downstream-only master this view will contain no rows.

* <tt>slot_name</tt>: An internal name for a given logical replication slot (a connection from a downstream master to this upstream master). This slot name is used by the downstream master to uniquely identify its self and is used with the <tt>pg_receivellog</tt> command when managing logical replication slots. The slot name is composed of the decoding plugin name, the upstream database oid, the downstream system identifier (from <tt>pg_control</tt>), the downstream slot number, and the downstream database oid.

* <tt>plugin</tt>: The logical replication plugin being used to decode WAL archives. You'll generally only see <tt>bdr_output</tt> here.

* <tt>database</tt>: The oid of the database being replicated by this slot. You can get the database name by joining on <tt>pg_database.oid</tt>.

* <tt>active</tt>: Whether this slot currently has an active connection.

* <tt>xmin</tt>: The lowest transaction ID this replication slot can "see", like the xmin of a transaction or prepared transaction. xmin should keep on advancing as replication continues.

* <tt>last_required_checkpoint</tt>: The checkpoint identifying the oldest WAL record required to bring this slot up to date with the upstream master. (This column is likely to be removed in a future version).

==== pg_stat_bdr ====

The <tt>pg_stat_bdr</tt> view is supplied by the <tt>bdr</tt> extension. It provides information on a server's connection(s) to its upstream master(s). It is not present on upstream-only masters.

The primary purpose of this view is to report statistics on the progress of LLSR apply on a per-upstream master connection basis.

View structure:

View "public.pg_stat_bdr"
Column | Type | Modifiers
--------------------+--------+-----------
rep_node_id | oid |
riremotesysid | name |
riremotedb | oid |
rilocaldb | oid |
nr_commit | bigint |
nr_rollback | bigint |
nr_insert | bigint |
nr_insert_conflict | bigint |
nr_update | bigint |
nr_update_conflict | bigint |
nr_delete | bigint |
nr_delete_conflict | bigint |
nr_disconnect | bigint |

Fields:

* <tt>rep_node_id</tt>: An internal identifier for the replication slot.

* <tt>riremotesysid</tt>: The remote database system identifier, as reported by the <tt>Database system identifier</tt> line of <tt>pg_controldata /path/to/datadir</tt>

* <tt>riremotedb</tt>: The remote database OID, ie the <tt>oid</tt> column of the remote server's <tt>pg_catalog.pg_database</tt> entry for the replicated database. You can get the database name with <tt>select datname from pg_database where oid = 12345</tt> (where '12345' is the <tt>riremotedb</tt> oid).

* <tt>rilocaldb </tt>: The local database OID, with the same meaning as <tt>riremotedb</tt> but with oids from the local system.

''The rest of the rows are statistics about this upstream master slot'':

* <tt>nr_commit</tt>: Number of commits applied to date from this master

* <tt>nr_rollback</tt>: Number of rollbacks performed by this apply process due to recoverable errors (deadlock retries, lost races, etc) or unrecoverable errors like mismatched constraint errors.

* <tt>nr_insert</tt>: Number of <tt>INSERT</tt>s performed

* <tt>nr_insert_conflict</tt>: Number of <tt>INSERT</tt>s that resulted in conflicts.

* <tt>nr_update</tt>: Number of <tt>UPDATE</tt>s performed

* <tt>nr_update_conflict</tt>: Number of <tt>UPDATE</tt>s that resulted in conflicts.

* <tt>nr_delete</tt>: Number of deletes performed

* <tt>nr_delete_conflict</tt>: Number of deletes that resulted in conflicts.

* <tt>nr_disconnect</tt>: Number of times this apply process has lost its connection to the upstream master since it was started.

This view does not contain any information about how far behind the upstream master this downstream master is. The upstream master's <tt>pg_stat_logical_replication</tt> and <tt>pg_stat_replication</tt> views must be queried to determine replication lag.

==== Monitoring replication status and lag ====

As with any replication setup, it is vital to monitor replication status on all BDR nodes to ensure no node is lagging severely behind the others or is stuck.

In the case of BDR a stuck or crashed node will eventually cause disk space and table bloat problems on other masters so stuck nodes should be detected and removed or repaired in a reasonably timely manner. Exactly how urgent this is depends on the workload of the BDR group.

The <tt>pg_stat_logical_replication</tt> view described above may be used to verify that a downstream master is connected to its upstream master - the <tt>active</tt> boolean column is <tt>t</tt> if there's a downstream master connected.

The <tt>xmin</tt> column provides an indication of whether replication is advancing; it should increase as replication progresses. There is no simple way to turn <tt>xmin</tt> into the time the last applied transaction was committed on the master, so it doesn't provide an indication of wall-clock lag.

To determine wall-clock replication lag an application-level ticker may be used to periodically update a timestamp in a replicated table. The difference between this timestamp on the upstream and downstream masters provides the wall-clock replication lag. For BDR one row may be added to the table for each BDR master, giving an indication of how much lag each master has relative to each other master.

=== Table and index usage statistics ===

Statistics on table and index usage are updated normally by the downstream master. This is essential for correct function of auto-vacuum. If there are no local writes on the downstream master and stats have not been reset these two views should show matching results between upstream and downstream:

* <tt>pg_stat_user_tables</tt>
* <tt>pg_statio_user_tables</tt>

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform <tt>UPDATE</tt>s and <tt>DELETE</tt>s than non-identifying indexes are.

The built-in index monitoring views are:

* <tt>pg_stat_user_indexes</tt>
* <tt>pg_statio_user_indexes</tt>

All these views are discussed in [http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE the PostgreSQL documentation on the statistics views].

=== Starting, stopping and managing replication ===

Replication is managed with the <tt>postgresql.conf</tt> settings described in "Parameter Reference" and "Configuration" above, and using the <tt>pg_receivellog</tt> utility command.

==== Starting a new LLSR connection ====

Logical replication is started automatically when a database is configured as a downstream master in <tt>postgresql.conf</tt> (see [[#Configuration|Configuration]]) and the postmaster is started. No explicit action is required to start replication, but replication will not actually work unless the upstream and downstream databases are identical within the requirements set by LLSR in the [[#Table definitions and DDL replication||Table definitions and DDL replication]] section.

<tt>pg_dump</tt> and <tt>pg_restore</tt> may be used to set up the new replica's database.

==== Viewing logical replication slots ====

Examining the state of logical replication is discussed in [[#Monitoring|Monitoring]].

==== Temporarily stopping an LLSR replica ====

LLSR replicas can be temporarily stopped by shutting down the downstream master's postmaster.

If the replica is not started back up before the upstream master discards the oldest WAL segment required for the downstream master to resume replay, as identified by the <tt>last_required_checkpoint</tt> column of <tt>pg_catalog.pg_stat_logical_replication</tt> then the replica will not resume replay. The error [[#Insufficient_WAL_segments_retained_.28.22requested_WAL_segment_..._has_already_been_removed.22.29|Insufficient WAL segments retained]] will be reported in the upstream master's logs. The replica must be re-created for replication to continue.

==== Removing an LLSR replica permanently ====

To remove a replication connection permanently, remove its entries from the downstream master's <tt>postgresql.conf</tt>, restart the downstream master, then use <tt>pg_receivellog</tt> to remove the replication slot on the upstream master.

It is important to remove the replication slot from the upstream master(s) to prevent xid wrap-around problems and issues with table bloat caused by delayed vacuum.

==== Cleaning up abandoned replication slots ====

To remove a replication slot that was used for a now-defunct replica, find its slot name from the <tt>[[#pg_stat_logical_replication|pg_stat_logical_replication]]</tt> view on the upstream master then run:

pg_receivellog -p 5434 -h master-hostname -d dbname \
--slot='bdr: 16384:5873181566046043070-1-16384:' --stop

where the argument to '--slot' is the slot name you found from the view.

You may need to do this if you've created and then deleted several replicas so <tt>max_logical_slots</tt> has filled up with entries for replicas that no longer exist.

== Bi-Directional Replication ==

Bi-Directional replication is built directly on LLSR by configuring two or more servers as both upstream ''and'' downstream masters of each other.

All of the Log Level Streaming Replication documentation applies to BDR and should be read before moving on to reading about and configuring BDR.

=== Bi-Directional Replication Use Cases ===

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

==== Simple multi-master pair ====

A simple mulit-master "HA Cluster" with two servers:

* Server "Alpha" - Master
* Server "Bravo" - Master

===== Configuration =====

Alpha:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="bravo"
bdr.bravo.dsn = 'dbname=dbtoreplicate'
track_commit_timestamp = on

Bravo:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="alpha"
bdr.alpha.dsn = 'dbname=dbtoreplicate'
track_commit_timestamp = on

See [[#Configuration|Configuration]] for an explanation of these parameters.

==== HA and Logical Standby ====
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

"HA Cluster":

* Server "Alpha" - Current Master
* Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
* Server "Charlie" - "Logical Standby" - downstream master

==== Very High Availability Multi-Master ====
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

==== 3-remote site simple Multi-Master Plex ====

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Alpha, Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Charlie using logical streaming replication

===== Configuration =====

If you wanted to test this configuration locally you could run three PostgreSQL instances on different ports. Such a configuration would look like the following if the port numbers were used as node names for the sake of notational clarity:

Config for node_5440:

port = 5440
bdr.connections='node_5441,node_5442'
bdr.node_5441.dsn='port=5441 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5440,node_5442'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440,node_5441'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5441.dsn='port=5441 dbname=postgres'

In a typical real-world configuration each server would be on the same port on a different host instead.

==== 3-remote site simple Multi-Master Circular Replication ====

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases. It's also less resilient to network disruptions and node faults.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

TODO: Regrettably this doesn't actually work yet because we don't cascade logical changes (yet).

===== Configuration =====

Using node names that match port numbers, for clarity

Config for node_5440:

port = 5440
bdr.connections='node_5441'
bdr.node_5441.dsn='port=5441 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5442'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440'
bdr.node_5440.dsn='port=5440 dbname=postgres'

This would usually be done in the real world with databases on different hosts, all running on the same port.

==== 3-remote site Max Availability Multi-Master Plex ====

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

Config left as an exercise for the reader.

==== N-site symmetric cluster replication ====

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

==== Complex/Assymetric Replication ====

Variety of options are possible.

=== Conflict Avoidance ===

==== Distributed Locking ====

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications as very low latency is critical for acceptable performance.

Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible but allow some types of conflict to occur and and resolve them when they arise.

==== Global Sequences ====

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

The SQL standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using <tt>DEFAULT nextval('mysequence')</tt>, as with PostgreSQL's <tt>SERIAL</tt> pseudo-type.

BDR requires sequences to work together across multiple nodes. This is implemented as a new <tt>SequenceAccessMethod</tt> API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

=== Conflict Detection & Resolution ===

Because local writes can occur on a master, conflict detection and avoidance is a concern for basic LLSR setups as well as full BDR configurations.

==== Lock Conflicts ====

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt> facility to look for issues with apply blocking on locks.

By concurrent activity on a row, we include

* explicit row level locking (<tt>SELECT ... FOR UPDATE/FOR SHARE</tt>)
* locking from foreign keys
* implicit locking because of row <tt>UPDATE</tt>s, <tt>INSERT</tt>s or <tt>DELETE</tt>s, either from local activity or apply from other servers

==== Data Conflicts ====

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in a consistent and idempotent manner so that all servers end up with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from <tt>pg_control</tt> though this may change in a future release.

<tt>UPDATE</tt>s and <tt>INSERT</tt>s may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur. The errors causing the conflict can be seen in the error log of the downstream master with the problem.

Updates which cannot locate a row are presumed to be <tt>DELETE</tt>/<tt>UPDATE</tt> conflicts. These are accepted as successful operations but in the case of <tt>UPDATE</tt> the data in the <tt>UPDATE</tt> is discarded.

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins. It is not practical to decide when a row should be merged and when a last-update-wins stragegy should be used at the database level; such decision making would require support for application-specific conflict resolution plugins.

Changing unlogged and logged tables in the same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

==== Examples ====

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

PgCon 2013 Developer Meeting

2013-05-13T19:16:24Z

Simon: /* Proposed Agenda Items */

A meeting of the most active PostgreSQL developers is being planned for Wednesday 22nd May, 2013 near the University of Ottawa, prior to pgCon 2013. In order to keep the numbers manageable, this meeting is '''by invitation only'''. Unfortunately it is quite possible that we've overlooked important code developers during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that this year the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.3 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies, unlike in previous years.

This is a PostgreSQL Community event. Room and refreshments/food sponsored by EnterpriseDB. Other companies sponsored attendance for their developers.

== Time & Location ==

The meeting will be from 8:30AM to 5PM, and will be in the "Red Experience" room at:

Novotel Ottawa
33 Nicholas Street
Ottawa
Ontario
K1N 9M7

Food and drink will be provided throughout the day, including breakfast from 8AM.

[http://maps.google.ca/maps?f=q&source=s_q&hl=en&geocode=&q=novotel+ottawa&aq=&sll=49.891235,-97.15369&sspn=36.237851,79.013672&ie=UTF8&hq=novotel+ottawa&hnear=&ll=45.421528,-75.683699&spn=0.036869,0.077162&z=14&iwloc=A&layer=c&cbll=45.425741,-75.689638&panoid=Z4FUGnkZkdHAOkIxyjjS9Q&cbp=12,25.83,,0,-0.6 View on Google Maps]

== Attendees ==

The following people have RSVPed to the meeting (in alphabetical order, by surname):

* Josh Berkus (secretary)
* Jeff Davis
* Andrew Dunstan
* Peter Eisentraut
* Dimitri Fontaine
* Andres Freund
* Stephen Frost
* Peter Geoghegan
* Kevin Grittner
* Robert Haas
* Magnus Hagander
* KaiGai Kohei
* Alexander Korotkov
* Tom Lane
* Fujii Masao
* Noah Misch
* Bruce Momjian
* Dave Page (chair)
* Simon Riggs

== Proposed Agenda Items ==

Please list proposed agenda items here:

* 9.4 Commitfest schedule
* [http://wiki.postgresql.org/wiki/Parallel_Query_Execution Parallel Query Execution] (Bruce, Noah)
* logical changeset generation review & integration (Andres)
* utilization of upcoming non-volatile RAM device (Kaigai)
* pluggable plan/exec nodes (Kaigai)
** to offload targetlist calculation, sorting, aggregates, ...
* [[GIN generalization]] (Alexander)
* An Extensibility Roadmap (dim)
* Representing severity - derive severity from SQLSTATE (Peter Geoghegan - see http://www.postgresql.org/message-id/CA+TgmoZEjq7va+SfDZQwk6E4emEWThENNyxfqEGhB3iuoT1OJw@mail.gmail.com)
* Error logging infrastructure - store normalized statistics about errors in a circular buffer (Peter Geoghegan). Arguably this could be discussed alongside SQLSTATE item.
* Failback with backup (Fujii Masao - related discussion is: http://www.postgresql.org/message-id/CAF8Q-Gxg3PQTf71NVECe-6OzRaew5pWhk7yQtbJgWrFu513s+Q@mail.gmail.com)
* Volume Management (Stephen Frost - wiki page will be forthcoming before the meeting)
* AXLE Project - Big data analytics for Postgres (Simon Riggs) - an overview of the feature plan, how project works and what community can expect

== Agenda ==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter
|- style="font-style:italic;background-color:lightgray;"
|08:00
|Breakfast
|

|- style="font-style:italic;background-color:lightgray;"
|08:30 - 08:45
|Welcome and introductions
|Dave Page

|-
|08:45 - 09:45
|Parallel Query Execution
|Bruce/Noah

|-
|09:45 - 10:30
|Logical changeset generation review & integration
|Andres

|- style="font-style:italic;background-color:lightgray;"
|10:30 - 10:45
|Coffee break
|

|-
|10:45 - 11:00
|Utilization of upcoming non-volatile RAM devices
|KaiGai

|-
|11:00 - 11:30
|Pluggable plan/exec nodes
|KaiGai

|-
|11:30 - 11:50
|Representing severity
|Peter G.

|-
|11:50 - 12:30
|Error logging infrastructure
|Peter G.

|- style="font-style:italic;background-color:lightgray;"
|12:30 - 13:30
|Lunch
|

|-
|13:30 - 14:15
|GIN generalization
|Alexander

|-
|14:15 - 15:00
|An Extensibility Roadmap
|Dimitri

|- style="font-style:italic;background-color:lightgray;"
|15:00 - 15:15
|Tea break
|

|-
|15:15 - 15:30
|9.4 Commitfest schedule
|All

|-
|15:30 - 16:45
|Goals, priorities, and resources for 9.4
|All

|- style="font-style:italic;background-color:lightgray;"
|16:45 - 17:00
|Any other business/group photo
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|17:00
|Finish
|
|}

PgCon 2013 Developer Meeting

2013-05-13T18:14:17Z

Simon: Aplha srot

A meeting of the most active PostgreSQL developers is being planned for Wednesday 22nd May, 2013 near the University of Ottawa, prior to pgCon 2013. In order to keep the numbers manageable, this meeting is '''by invitation only'''. Unfortunately it is quite possible that we've overlooked important code developers during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that this year the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.3 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies, unlike in previous years.

This is a PostgreSQL Community event. Room and refreshments/food sponsored by EnterpriseDB. Other companies sponsored attendance for their developers.

== Time & Location ==

The meeting will be from 8:30AM to 5PM, and will be in the "Red Experience" room at:

Novotel Ottawa
33 Nicholas Street
Ottawa
Ontario
K1N 9M7

Food and drink will be provided throughout the day, including breakfast from 8AM.

[http://maps.google.ca/maps?f=q&source=s_q&hl=en&geocode=&q=novotel+ottawa&aq=&sll=49.891235,-97.15369&sspn=36.237851,79.013672&ie=UTF8&hq=novotel+ottawa&hnear=&ll=45.421528,-75.683699&spn=0.036869,0.077162&z=14&iwloc=A&layer=c&cbll=45.425741,-75.689638&panoid=Z4FUGnkZkdHAOkIxyjjS9Q&cbp=12,25.83,,0,-0.6 View on Google Maps]

== Attendees ==

The following people have RSVPed to the meeting (in alphabetical order, by surname):

* Josh Berkus (secretary)
* Jeff Davis
* Andrew Dunstan
* Peter Eisentraut
* Dimitri Fontaine
* Andres Freund
* Stephen Frost
* Peter Geoghegan
* Kevin Grittner
* Robert Haas
* Magnus Hagander
* KaiGai Kohei
* Alexander Korotkov
* Tom Lane
* Fujii Masao
* Noah Misch
* Bruce Momjian
* Dave Page (chair)
* Simon Riggs

== Proposed Agenda Items ==

Please list proposed agenda items here:

* 9.4 Commitfest schedule
* [http://wiki.postgresql.org/wiki/Parallel_Query_Execution Parallel Query Execution] (Bruce, Noah)
* logical changeset generation review & integration (Andres)
* utilization of upcoming non-volatile RAM device (Kaigai)
* pluggable plan/exec nodes (Kaigai)
** to offload targetlist calculation, sorting, aggregates, ...
* [[GIN generalization]] (Alexander)
* An Extensibility Roadmap (dim)
* Representing severity - derive severity from SQLSTATE (Peter Geoghegan - see http://www.postgresql.org/message-id/CA+TgmoZEjq7va+SfDZQwk6E4emEWThENNyxfqEGhB3iuoT1OJw@mail.gmail.com)
* Error logging infrastructure - store normalized statistics about errors in a circular buffer (Peter Geoghegan). Arguably this could be discussed alongside SQLSTATE item.
* Failback with backup (Fujii Masao - related discussion is: http://www.postgresql.org/message-id/CAF8Q-Gxg3PQTf71NVECe-6OzRaew5pWhk7yQtbJgWrFu513s+Q@mail.gmail.com)
* Volume Management (Stephen Frost - wiki page will be forthcoming before the meeting)

== Agenda ==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter
|- style="font-style:italic;background-color:lightgray;"
|08:00
|Breakfast
|

|- style="font-style:italic;background-color:lightgray;"
|08:30 - 08:45
|Welcome and introductions
|Dave Page

|-
|08:45 - 09:45
|Parallel Query Execution
|Bruce/Noah

|-
|09:45 - 10:30
|Logical changeset generation review & integration
|Andres

|- style="font-style:italic;background-color:lightgray;"
|10:30 - 10:45
|Coffee break
|

|-
|10:45 - 11:00
|Utilization of upcoming non-volatile RAM devices
|KaiGai

|-
|11:00 - 11:30
|Pluggable plan/exec nodes
|KaiGai

|-
|11:30 - 11:50
|Representing severity
|Peter G.

|-
|11:50 - 12:30
|Error logging infrastructure
|Peter G.

|- style="font-style:italic;background-color:lightgray;"
|12:30 - 13:30
|Lunch
|

|-
|13:30 - 14:15
|GIN generalization
|Alexander

|-
|14:15 - 15:00
|An Extensibility Roadmap
|Dimitri

|- style="font-style:italic;background-color:lightgray;"
|15:00 - 15:15
|Tea break
|

|-
|15:15 - 15:30
|9.4 Commitfest schedule
|All

|-
|15:30 - 16:45
|Goals, priorities, and resources for 9.4
|All

|- style="font-style:italic;background-color:lightgray;"
|16:45 - 17:00
|Any other business/group photo
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|17:00
|Finish
|
|}

BDR User Guide

2013-05-08T07:04:59Z

Simon:

----
This page is the users and administrators guide for BDR. If you're looking for technical details on the project plan and implementation, see [[BDR Project]].
----

= BDR User Guide =

BDR (BiDrectional Replication) is a feature being developed for inclusion in PostgreSQL core that provides greatly enhanced replication capabilities.

BDR allows users to create a geographically distributed multi-master database using Logical Log Streaming Replication (LLSR) transport.
BDR is designed to provide both high availability and geographically distributed disaster recovery capabilities.

BDR is not “clustering” as some vendors use the term, in that it doesn't have a distributed lock manager, global transaction co-ordinator, etc. Each member server is separate yet connected, with design choices that allow separation between nodes that would not be possible with global transaction coordination.

Guidance on getting a testing setup established are in [[#Initial setup]]. Please read the full documentation if you intend to put BDR into production.

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows one PostgreSQL master (the "upstream master") to stream a sequence of changes to another read/write PostgreSQL server (the "downstream master"). Data is sent in one direction only over a normal libpq connection.

Multiple LLSR connections can be used to set up bi-directional replication as discussed later in this guide.

=== Overview of logical replication ===

In some ways LLSR is similar to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective; both replicate changes from one server to another. However, in LLSR the receiving server is also a full master database that can make changes, unlike the read-only replicas offered by PLSR hot standby. Additionally, LLSR is per-database, whereas PLSR is per-cluster and replicates all databases at once. There are many more differences discussed in the relevant sections of this document.

In LLSR the data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after <tt>CREATE DATABASE</tt>. A restart of the downstream master is also required. The upstream master only needs restarting if the <tt>max_logical_slots</tt> parameter is too low to allow a new replica to be added. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated. Setup is discussed in more detail below.

Changes are processed by the downstream master using <tt>bdr</tt> plug-ins. This allows flexible handing of replication input, including:

* BDR apply process - applies logical changes to the downstream master. The apply process makes changes directly rather than generating SQL text and then parse/plan/executing SQL.
* Textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* <tt>pg_xlogdump</tt> - examines physical WAL records and produces textual debugging output. This server program is included in PostgreSQL 9.3.

=== Replication of DML changes ===

All changes are replicated: <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> and <tt>TRUNCATE</tt>.

(TRUNCATE is not yet implemented, but will be implemented before the feature goes to final release).

Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though it has overheads that mean that it doesn't always use less bandwidth than PLSR.

Locks taken by <tt>LOCK</tt> and <tt>SELECT ... FOR UPDATE/SHARE</tt> on the upstream master are not replicated to downstream masters. Locks taken automatically by <tt>INSERT</tt>, <tt>UPDATE</tt>, <tt>DELETE</tt> or <tt>TRUNCATE</tt> *are* taken on the downstream master and may delay replication apply or concurrent transactions - see [[#Lock Conflicts|Lock Conflicts]].

<tt>TEMPORARY</tt> and <tt>UNLOGGED</tt> tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

<tt>DELETE</tt> and <tt>UPDATE</tt> statements that affect multiple rows on upstream master will cause a series of row changes on downstream master. These are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. <tt>INSERT</tt> on upstream master do not require a unique constraint in order to replicate correctly. <tt>UPDATE</tt>s and <tt>DELETE</tt>s require some form of unique constraint, either <tt>PRIMARY KEY</tt> or <tt>UNIQUE NOT NULL</tt>. A warning is issued in the downstream master's logs if the expected constraint is absent.

<tt>UPDATE</tt>s that change the value of the Primary Key of a table will be replicated correctly.

The values applied are the final values from the <tt>UPDATE</tt> on the upstream master, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value. Volatile or stable functions are evaluated on the master side and the resulting values are replicated. Consequently any function side-effects (writing files, network socket activity, updating internal PostgreSQL variables, etc) will not occur on the replicas as the functions are not run again on the replica.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master (see "Limitations").

The current LLSR plugin implementation uses the binary libpq protocol, so it requires that the upstream and downstream master use same CPU architecture and word-length, i.e. "identical servers", as with physical replication. A textual output option will be added later for passing data between non-identical servers, e.g. laptops or mobile devices communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is efficiently implemented. Parallel apply is a possible future feature, especially for changes made while holding <tt>AccessExclusiveLock</tt>.

Changes are applied to the downstream master in the sequence in which they were commited on the upstream master. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions spill to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

<tt>SET</tt> statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. We always update the correct tables, whatever the setting of <tt>search_path</tt>. Values are replicated correctly irrespective of the values of <tt>bytea_output</tt>, <tt>TimeZone</tt>, <tt>DateStyle</tt>, etc.

<tt>NOTIFY</tt> is not supported across log based replication, either physical or logical. <tt>NOTIFY</tt> and <tt>LISTEN</tt> will work fine on the upstream master but an upstream <tt>NOTIFY</tt> will not trigger a downstream <tt>LISTEN</tt>er.

In some cases, additional deadlocks can occur on apply. This causes an automatic retry of the apply of the replaying transaction and is only an issue if the deadlock recurs repeatedly, delaying replication.

From a performance and concurrency perspective the BDR apply process is similar to a normal backend. Frequent conflicts with locks from other transactions when replaying changes can slow things down and thus increase replication delay, so reducing the frequency of such conflicts can be a good way to speed things up. Any lock held by another transaction on the downstream master - <tt>LOCK</tt> statements, <tt>SELECT ... FOR UPDATE/FOR SHARE</tt>, or <tt>INSERT</tt>/<tt>UPDATE</tt>/<tt>DELETE</tt> row locks - can delay replication if the replication apply process needs to change the locked table/row.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching <tt>"Schemaname"."Tablename"</tt> on both upstream and downstream masters. e.g. changes from upstream's <tt>public.mytable</tt> will go to downstream's <tt>public.mytable</tt> while changes to the upstream <tt>mychema.mytable</tt> will go to the downstream <tt>myschema.mytable</tt>. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful synchronization of table definitions on each node otherwise <tt>ERROR</tt>s will be generated by the replication apply process. In general, tables must be an exact match between upstream and downstream masters.

There are no plans to implement working replication between dissimilar table definitions.

Tables must meet the following requirements to be compatible for purposes of LLSR:

* The downstream master must only have constraints (<tt>CHECK</tt>, <tt>UNIQUE</tt>, <tt>EXCLUSION</tt>, <tt>FOREIGN KEY</tt>, etc) that are also present on the upstream master. Replication may initially work with mismatched constraints but is likely to fail as soon as the downstream master rejects a row the upstream master accepted.
* The table referenced by a FOREIGN KEY on a downstream master must have all the keys present in the upstream master version of the same table.
* Storage parameters must match except for as allowed below
* Inheritance must be the same
* Dropped columns on master must be present on replicas
* Custom types and enum definitions must match exactly
* Composite types and enums must have the same oids on master and replication target
* Extensions defining types used in replicated tables must be of the same version or fully SQL-level compatible and the oids of the types they define must match.

The following differences are permissible between tables on different nodes:

* The table's <tt>pg_class</tt> oid, the oid of its associated TOAST table, and the oid of the table's rowtype in <tt>pg_type</tt> may differ;
* Extra or missing non-<tt>UNIQUE</tt> indexes
* Extra keys in downstream lookup tables for <tt>FOREIGN KEY</tt> references that are not present on the upstream master
* The table-level storage parameters for fillfactor and autovacuum
* Triggers and rules may differ (they are not executed by replication apply)

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR (see [[#LLSR Limitations|LLSR Limitations]]).

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of <tt>session_replication_role = origin</tt>.

In future it is expected that composite types and enums with non-identical oids will be converted using text output and input functions. This feature is not yet implemented.

=== Parameter Reference ===

* <tt>bdr.connections</tt> - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

* <tt>bdr.<nodename>.dsn</tt> - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

* <tt>max_logical_slots</tt> - LLSR uses persistent slots in memory which are reserved for each node at server start

* <tt>wal_level</tt> - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== LLSR limitations ===

Table definitions, types, extensions, etc must be near identical between upstream and downstream masters. See [[#Table definitions and DDL replication|Table definitions and DDL replication]].

DDL replication is not yet supported.

No feedback from downstream masters to the upstream master is implemented for asynchronous LLSR, so upstream masters must be configured to keep enough WAL. See [[#Configuration|Configuration]].

=== Initial setup ===

To set up LLSR or BDR you first need a patched PostgreSQL that can support LLSR/BDR, then you need to create one or more LLSR/BDR senders and one or more LLSR/BDR receivers.

==== Installing the patched PostgreSQL binaries ====

Currently BDR is only available in builds of the 'bdr' branch on Andres Freund's git repo on git.postgresql.org. PostgreSQL 9.2 and below do not support BDR, and 9.3 requires patches, so this guide will not work for you if you are trying to use a normal install of PostgreSQL.

First you need to clone, configure, compile and install like normal. Clone the sources from <tt>git://git.postgresql.org/git/users/andresfreund/postgres.git</tt> and checkout the <tt>bdr</tt> branch.

If you have an existing local PostgreSQL git tree specify it as <tt>--reference /path/to/existing/tree</tt> to greatly speed your git clone.

Example:

mkdir -p $HOME/bdr
cd bdr
git clone git://git.postgresql.org/git/users/andresfreund/postgres.git $HOME/bdr/postgres-bdr-src
cd postgres-bdr-src
./configure --prefix=$HOME/bdr/postgres-bdr-bin
make install
cd contrib/bdr
make install

This will put everything in <tt>$HOME/bdr</tt>, with the source code and build tree in <tt>$HOME/bdr/postgres-bdr-src</tt> and the installed PostgreSQL in <tt>$HOME/bdr/postgres-bdr-bin</tt>. This is a convenient setup for testing and development because it doesn't require you to set up new users, wrangle permissions, run anything as root, etc, but it isn't recommended that you deploy this way in production.

To actually use these new binaries you will need to:

export PATH=$HOME/bdr/postgres-bdr-bin/bin:$PATH

before running <tt>initdb</tt>, <tt>postgres</tt>, etc. You don't have to use the <tt>psql</tt> or <tt>libpq</tt> you compiled but you're likely to get version mismatch warnings if you don't.

=== Configuration ===

The configuration for a simple single-master to single-replica configuration looks like:

Upstream (sender) <tt>postgresql.conf</tt>:

wal_level = 'logical' # Include enough info for logical replication
max_logical_slots = X # Number of LLSR senders + any receivers
max_wal_senders = Y # Y = max_logical_slots plus any physical
# streaming requirements
wal_keep_segments = 5000 # Master must retain enough WAL segments to let
# replicas catch up. Correct value depends on
# rate of writes on master, max replica downtime
# allowable. 5000 segments requires 78GB
# in pg_xlog

Downstream (receiver) <tt>postgresql.conf</tt>:

shared_preload_libraries = 'bdr'

bdr.connections="name_of_upstream_master" # list of upstream master nodenames
bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection
# from downstream to upstream master
bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case
# where the databasename on upstream
# and downstream master differ.
# (Not yet implemented)
bdr.<nodename>.apply_delay # optional parameter to delay apply of
# transactions, time in milliseconds
bdr.synchronous_commit = ...; # optional parameter to set the
# synchronous_commit parameter the
# apply processes will be using
max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network and for heavy bursts of write activity on the master. Keep in mind that enough disk space must be available for the WAL segments, each of which is 16MB. See "Insufficient WAL segments retained".

Note that a server can be both sender and receiver, either two servers to each other or more complex configurations like replication chains/trees.

The upstream (sender) <tt>pg_hba.conf</tt> must be configured to allow the downstream master to connect for replication. Otherwise you'll see errors like the following on the downstream master:

FATAL: could not connect to the primary server: FATAL: no pg_hba.conf entry for replication connection from host "[local]", user "postgres"

A suitable <tt>pg_hba.conf</tt> entry for a replication connection from the replica server 10.1.4.8 might be:

host replication postgres 10.1.4.8/32 trust

(the user name should match the user name configured in the downstream master's dsn. md5 password authentication is supported.)

For more details on these parameters, see [[#Parameter Reference|Parameter Reference]].

=== Troubleshooting ===

==== Could not access file "bdr": No such file or directory ====

If you see the error:

FATAL: could not access file "bdr": No such file or directory

when starting a database set up to receive BDR replication, you probably forgot to install <tt>contrib/bdr</tt>. See above.

==== Invalid value for parameter ====

An error like:

LOG: invalid value for parameter ...

when setting one of these parameters means your server doesn't support logical replication and will need to be patched or updated.

==== Insufficient WAL segments retained ("requested WAL segment ... has already been removed") ====

If <tt>wal_keep_segments</tt> is insufficient to meet the requirements of a replica that has fallen far behind, the master will report errors like:

ERROR: requested WAL segment 00000001000000010000002D has already been removed

Currently the replica errors look like:

WARNING: Starting logical replication
LOG: data stream ended
LOG: worker process: master (PID 23812) exited with exit code 0
LOG: starting background worker process "master"
LOG: master initialized on master, remote dbname=master port=5434 replication=true fallback_application_name=bdr
LOG: local sysid 5873181566046043070, remote: 5873181102189050714
LOG: found valid replication identifier 1
LOG: starting up replication at 1 from 1/2D9CA220

but a more explicit error message for this condition is planned.

The only way to recover from this fault is to re-seed the replica database.

This fault could be prevented with feedback from the replica to the master, but this feature is not planned for the first release of BDR. Another alternative considered for future releases is making wal_keep_segments a dynamic parameter that is sized on demand.

Monitoring of maximum replica lag and appropriate adjustment of wal_keep_segments will prevent this fault from arising.

==== Couldn't find logical slot ====

An error like:

ERROR: couldn't find logical slot "bdr: 16384:5873181566046043070-1-24596:"

on the upstream master suggests that a downstream master is trying to connect to a logical replication slot that no longer exists. The slot can not be re-created, so it is necessary to re-seed the downstream replica database.

=== Operational Issues and Debugging ===

In LLSR there are no user-level (ie SQL visible) ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

The following views are available for monitoring replication activity:

* <tt>[http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE pg_stat_replication]</tt>
* <tt>pg_stat_logical_replication</tt> (described below)
* <tt>pg_stat_bdr</tt> (described below)

The following configuration and logging parameters are useful for monitoring replication:

* <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt>

==== pg_stat_logical_replication ====

The new <tt>pg_stat_logical_replication</tt> view is specific to logical replication. It is based on the underlying <tt>pg_stat_get_logical_replication_slots</tt> function and has the following structure:

View "pg_catalog.pg_stat_logical_replication"
Column | Type | Modifiers
--------------------------+---------+-----------
slot_name | text |
plugin | text |
database | oid |
active | boolean |
xmin | xid |
last_required_checkpoint | text |

It contains one row for every connection from a downstream master to the server being queried (the upstream master).

* <tt>slot_name</tt>: An internal name for a given logical replication slot (a connection from a downstream master to this upstream master). This slot name is used by the downstream master to uniquely identify its self and is used with the <tt>pg_receivellog</tt> command when managing logical replication slots.

* <tt>plugin</tt>: The logical replication plugin being used to decode WAL archives. You'll generally only see <tt>bdr_output</tt> here.

* <tt>database</tt>: The oid of the database being replicated by this slot. You can get the database name by joining on <tt>pg_database.oid</tt>.

* <tt>active</tt>: Whether this slot currently has an active connection.

* <tt>xmin</tt>: The lowest transaction ID this replication slot can "see". (TODO)

* <tt>last_required_checkpoint</tt>: The checkpoint identifying the oldest WAL record required to bring this slot up to date with the upstream master. (TODO)

==== pg_stat_bdr ====

The <tt>pg_catalog.pg_stat_bdr</tt> view ... (TODO)

View structure:

(TODO)

=== Table and index usage statistics ===

Statistics on table and index usage are updated normally by the downstream master. This is essential for correct function of auto-vacuum. If there are no local writes on the downstream master and stats have not been reset these two views should show matching results between upstream and downstream:

* <tt>pg_stat_user_tables</tt>
* <tt>pg_statio_user_tables</tt>

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform <tt>UPDATE</tt>s and <tt>DELETE</tt>s than non-identifying indexes are.

The built-in index monitoring views are:

* <tt>pg_stat_user_indexes</tt>
* <tt>pg_statio_user_indexes</tt>

All these views are discussed in [http://www.postgresql.org/docs/current/static/monitoring-stats.html#MONITORING-STATS-VIEWS-TABLE the PostgreSQL documentation on the statistics views].

=== Starting, stopping and managing replication ===

TODO: Extension to improve this?

==== Starting a new LLSR connection ====

Logical replication is started automatically when a database is configured as a downstream master in <tt>postgresql.conf</tt> (see [[#Configuration|Configuration]]) and the postmaster is started. No explicit action is required to start replication, but replication will not actually work unless the upstream and downstream databases are identical within the requirements set by LLSR in the [[#Table definitions and DDL replication||Table definitions and DDL replication]] section.

==== Viewing logical replication slots ====

Examining the state of logical replication is discussed in [[#Monitoring|Monitoring]].

==== Temporarily stopping an LLSR replica ====

LLSR replicas can be temporarily stopped by shutting down the downstream master's postmaster.

If the replica is not started back up before the upstream master discards the oldest WAL segment required for the downstream master to resume replay, as identified by the <tt>last_required_checkpoint</tt> column of <tt>pg_catalog.pg_stat_logical_replication</tt> then the replica will not resume replay. The error [[#Insufficient_WAL_segments_retained_.28.22requested_WAL_segment_..._has_already_been_removed.22.29|Insufficient WAL segments retained]] will be reported in the upstream master's logs. The replica must be re-seeded for replication to continue.

TODO: Discuss any SQL-level, per-database functions for managing replication.

==== Removing an LLSR replica permanently ====

To remove a replication connection permanently, remove its entries from the downstream master's <tt>postgresql.conf</tt>, restart the downstream master, then use <tt>pg_receivellog</tt> to remove the replication slot on the upstream master.

TODO pending merge of downstream control functions.

==== Cleaning up abandoned replication slots ====

To remove a replication slot that was used for a now-defunct replica, find its slot name from the <tt>[[#pg_stat_logical_replication|pg_stat_logical_replication]]</tt> view on the upstream master then run:

pg_receivellog -p 5434 -h master-hostname -d dbname \
--slot='bdr: 16384:5873181566046043070-1-16384:' --stop

where the argument to '--slot' is the slot name you found from the view.

You may need to do this if you've created and then deleted several replicas so <tt>max_logical_slots</tt> has filled up with entries for replicas that no longer exist.

== Bi-Directional Replication ==

Bi-Directional replication is built directly on LLSR by configuring two or more servers as both upstream ''and'' downstream masters of each other.

All of the Log Level Streaming Replication documentation applies to BDR and should be read before moving on to reading about and configuring BDR.

=== Bi-Directional Replication Use Cases ===

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

==== Simple multi-master pair ====

A simple mulit-master "HA Cluster" with two servers:

* Server "Alpha" - Master
* Server "Bravo" - Master

===== Configuration =====

Alpha:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="bravo"
bdr.bravo.dsn = 'dbname=dbtoreplicate'

Bravo:

wal_level = 'logical'
max_logical_slots = 3
max_wal_senders = 4
wal_keep_segments = 5000
shared_preload_libraries = 'bdr'
bdr.connections="alpha"
bdr.alpha.dsn = 'dbname=dbtoreplicate'

See [[#Configuration|Configuration]] for an explanation of these parameters.

==== HA and Logical Standby ====
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

"HA Cluster":

* Server "Alpha" - Current Master
* Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
* Server "Charlie" - "Logical Standby" - downstream master

==== Very High Availability Multi-Master ====
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

==== 3-remote site simple Multi-Master Plex ====

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Alpha, Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Charlie using logical streaming replication

===== Configuration =====

If you wanted to test this configuration locally you could run three PostgreSQL instances on different ports. Such a configuration would look like the following if the port numbers were used as node names for the sake of notational clarity:

Config for node_5440:

port = 5440
bdr.connections='node_5441,node_5442'
bdr.node_5441.dsn='port=5441 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5440,node_5442'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440,node_5441'
bdr.node_5440.dsn='port=5440 dbname=postgres'
bdr.node_5441.dsn='port=5441 dbname=postgres'

In a typical real-world configuration each server would be on the same port on a different host instead.

==== 3-remote site simple Multi-Master Circular Replication ====

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases. It's also less resilient to network disruptions and node faults.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

TODO: Regrettably this doesn't actually work yet because we don't cascade logical changes (yet).

===== Configuration =====

Using node names that match port numbers, for clarity

Config for node_5440:

port = 5440
bdr.connections='node_5441'
bdr.node_5441.dsn='port=5441 dbname=postgres'

Config for node_5441:

port = 5441
bdr.connections='node_5442'
bdr.node_5442.dsn='port=5442 dbname=postgres'

Config for node_5442:

port = 5442
bdr.connections='node_5440'
bdr.node_5440.dsn='port=5440 dbname=postgres'

This would usually be done in the real world with databases on different hosts, all running on the same port.

==== 3-remote site Max Availability Multi-Master Plex ====

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

Config left as an exercise for the reader.

==== N-site symmetric cluster replication ====

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

==== Complex/Assymetric Replication ====

Variety of options are possible.

=== Conflict Avoidance ===

==== Distributed Locking ====

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications as very low latency is critical for acceptable performance.

Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible but allow some types of conflict to occur and and resolve them when they arise.

==== Global Sequences ====

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

The SQL standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using <tt>DEFAULT nextval('mysequence')</tt>, as with PostgreSQL's <tt>SERIAL</tt> pseudo-type.

BDR requires sequences to work together across multiple nodes. This is implemented as a new <tt>SequenceAccessMethod</tt> API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

=== Conflict Detection & Resolution ===

Because local writes can occur on a master, conflict detection and avoidance is a concern for basic LLSR setups as well as full BDR configurations.

==== Lock Conflicts ====

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the <tt>[http://www.postgresql.org/docs/current/static/runtime-config-logging.html#GUC-LOG-LOCK-WAITS log_lock_waits]</tt> facility to look for issues with apply blocking on locks.

By concurrent activity on a row, we include

* explicit row level locking (<tt>SELECT ... FOR UPDATE/FOR SHARE</tt>)
* locking from foreign keys
* implicit locking because of row <tt>UPDATE</tt>s, <tt>INSERT</tt>s or <tt>DELETE</tt>s, either from local activity or apply from other servers

==== Data Conflicts ====

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in a consistent and idempotent manner so that all servers end up with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from <tt>pg_control</tt> though this may change in a future release.

<tt>UPDATE</tt>s and <tt>INSERT</tt>s may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur. The errors causing the conflict can be seen in the error log of the downstream master with the problem.

Updates which cannot locate a row are presumed to be <tt>DELETE</tt>/<tt>UPDATE</tt> conflicts. These are accepted as successful operations but in the case of <tt>UPDATE</tt> the data in the <tt>UPDATE</tt> is discarded.

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins. It is not practical to decide when a row should be merged and when a last-update-wins stragegy should be used at the database level; such decision making would require support for application-specific conflict resolution plugins.

Changing unlogged and logged tables in the same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

==== Examples ====

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

GSoC 2013

2013-04-08T21:04:39Z

Simon: /* Project Ideas */

== Projects ==

The GSoC projects for 2013 will be listed here when selected:

#

== What is GSoC? ==

Google Summer of Code (GSoC) is a global program that offers student developers stipends to write code for various open source software projects. We have worked with several open source, free software, and technology-related groups to identify and fund several projects over a three month period. Since its inception in 2005, the program has brought together over 4,500 students and more than more than 4,000 mentors & co-mentors from over 85 countries worldwide, all for the love of code. Through Google Summer of Code, accepted student applicants are paired with a mentor or mentors from the participating projects, thus gaining exposure to real-world software development scenarios and the opportunity for employment in areas related to their academic pursuits. In turn, the participating projects are able to more easily identify and bring in new developers. Best of all, more source code is created and released for the use and benefit of all.

PostgreSQL has an official summer of code page: http://www.postgresql.org/developer/summerofcode.html

== Advice for Students ==

We have developed the following [http://www.postgresql.org/developer/summerofcodeadvice Advice for Students] page to get you started.

Also, students who discuss their proposals with Postgres project members *before* the application deadline are much more likely to be successful.

== Mailing list for student questions ==

For GSoC program questions and discussion, please subscribe to:

http://archives.postgresql.org/pgsql-students/

We can have non-code related discussions on this list, and help answer questions about proposal writing.

Discussion of code should be done on the list for the specific project a student is working on.

== IRC ==

We have an IRC channel at #postgresql and you are welcome to answer questions. Our GSoC Admins are: darkixion and agliodbs. Do not DM them without asking first!

You are welcome to ping us in the channel, or ask the channel general questions about a proposal. If you do not find help in the channel, feel free to send an email to pgsql-students@postgresql.org for help.

== Proposal Format ==

Students are responsible for writing a proposal and submitting it to Google before the application deadline. The following outline was adapted from the [http://www.perlfoundation.org/how_to_write_a_proposal Perl Foundation open source proposal HOWTO]. A strong proposal will include:

* Project Title
* Name of proposer and email
* Synopsis
* Benefits to the PostgreSQL Community
* Quantifiable results
* Project Details
* Inch-stones (project broken into small, distinct chunks)
* Project Schedule
* Completeness Criteria
* Bio
**Blog
**Github
**@Twitter

== Project Ideas ==

Project ideas are to be added here.

(Please can we have visibility of ideas on Hackers please to avoid overreaching what is possible in the time, and also working on dubious projects.)

=== Core ===
* UPDATE ... RETURNING OLD [http://www.postgresql.org/message-id/20130218171259.GA26999@fetter.org link]
* Add RETURNING to DDL (CREATE, ALTER, DROP) and possibly DCL (GRANT, REVOKE) [http://www.postgresql.org/message-id/20130218171259.GA26999@fetter.org link]

=== Extensions ===
* cube extension improvements (indexing, type support, new KNN search metrics) [http://www.postgresql.org/message-id/6A7E75B1-64DD-4F5F-A991-435E3E5A24FB@gmail.com link]

=== Tools ===
* Rewrite (add) pg_dump and pg_restore utilities as libraries (.so, .dll & .dylib) [http://www.postgresql.org/message-id/1811491181.20130215163950@gf.microolap.com link]
* Extending MADlib functions to fill in (extrapolate) missing values in data sets [http://www.postgresql.org/message-id/B654BEBE-32D9-4670-BBB7-BC846AE5B785@gmail.com link1] [http://www.postgresql.org/message-id/511E7193.4020907@agliodbs.com link2]
* pg_upgrade support for Debian's pg_upgradecluster [http://www.postgresql.org/message-id/20130218213711.GA1005@awork2.anarazel.de link]

== Project Admins ==

* Thom Brown
* Josh Berkus

== 2013 Mentors ==

Mentors volunteered who have been active on -hackers list:
* Alvaro Herrera
* Stephen Frost (maybe)
* Dimitri Fontaine
* Alexander Korotkov
* Pavel Golub
* David Fetter
* Magnus Hagander
* Christoph Berg
* Tomas Vondra

Other volunteers who can potentially act as assitants to mentors:
* Atri Sharma
* Gilberto Castillo

== Past Success ==

Need to add

== GOALS: ==
* usable code
** useful/novel ideas
** research projects
* longer term contributors

== TODOs: ==

* Kick-off Meeting for Community Members
* Update GSOC page
* Advertising?
* Blog that we're participating and seeking students
* Round of private emails to people who have participated in the past: Heikki, Simon, Mark, Stephen, Merlin
** request interest, and then follow up in asking about possible topics for students
* Mentor recruitment and then email to -hackers
** do this much later when we have some proposals in?

* Recruitment -- no organized group effort?
** -announce, -general, -hackers
** user group lists
** phppgadmin/pgadmin
** berkeley
** Univ. of Maryland -- contact them?

* Identify the commitfest that the code will be submitted to

== Expectations ==

* Stuff to keep students together:
** Regular blogging from students
** weekly group IRC checkin? -- two checkin times maybe?

* Have students communicate on -hackers where appropriate (didn't really work?)
** Or other relevant -devel lists

* Mailing list
** pgsql-students (?) vs. -hackers (?) maybe up to mentor?
** mentors mailing list -admin mailing list, berkus said?
** students mailing list via gsoc

[[Category:Google Summer of Code]]

BDR User Guide

2013-03-12T16:04:19Z

Simon: /* 3-remote site simple Multi-Master Circular Replication */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master Plex ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Alpha, Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Charlie using logical streaming replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5441:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5442:
** port = 5442
** bdr.connections='node_5440,node_5441'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site simple Multi-Master Circular Replication ===

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

* config for 5441:
** port = 5441
** bdr.connections='node_5442'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5442:
** port = 5442
** bdr.connections='node_5440'
** bdr.node_5440.dsn='port=5440 dbname=postgres'

Regrettably this doesn't actually work yet because we don't cascade logical changes (yet).

=== 3-remote site Max Availability Multi-Master Plex ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

Config left as an exercise for the reader.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

Changing unlogged and logged tables in same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T16:01:53Z

Simon: /* 3-remote site Max Availability Multi-Master Plex */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master Plex ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Alpha, Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Charlie using logical streaming replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5441:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5442:
** port = 5442
** bdr.connections='node_5440,node_5441'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site simple Multi-Master Circular Replication ===

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

* config for 5441:
** port = 5441
** bdr.connections='node_5442'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5442:
** port = 5442
** bdr.connections='node_5440'
** bdr.node_5440.dsn='port=5440 dbname=postgres'

=== 3-remote site Max Availability Multi-Master Plex ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

Config left as an exercise for the reader.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

Changing unlogged and logged tables in same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T16:01:12Z

Simon: /* 3-remote site simple Multi-Master Plex */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master Plex ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Alpha, Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Charlie using logical streaming replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5441:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5442:
** port = 5442
** bdr.connections='node_5440,node_5441'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site simple Multi-Master Circular Replication ===

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

* config for 5441:
** port = 5441
** bdr.connections='node_5442'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5442:
** port = 5442
** bdr.connections='node_5440'
** bdr.node_5440.dsn='port=5440 dbname=postgres'

=== 3-remote site Max Availability Multi-Master Plex ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

Changing unlogged and logged tables in same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T15:59:35Z

Simon: /* 3-remote site simple Multi-Master Circular Replication */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master Plex ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site simple Multi-Master Circular Replication ===

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases.

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming replication

* Site 2
** Server "Charlie" - Master - feeds changes to Echo using logical streaming replication

* Site 3
** Server "Echo" - Master - feeds changes to Alpha using logical streaming replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

* config for 5441:
** port = 5441
** bdr.connections='node_5442'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5442:
** port = 5442
** bdr.connections='node_5440'
** bdr.node_5440.dsn='port=5440 dbname=postgres'

=== 3-remote site Max Availability Multi-Master Plex ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

Changing unlogged and logged tables in same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T15:55:14Z

Simon: /* Bi-Directional Replication Use Cases */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master Plex ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site simple Multi-Master Circular Replication ===

Simpler config uses "circular replication". This is simpler but results in higher latency for changes as the number of nodes increases.

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master Plex ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

Changing unlogged and logged tables in same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T15:49:10Z

Simon: /* Data Conflicts */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors because of primary keys, unique indexes and exclusion constraints when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

Changing unlogged and logged tables in same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T15:45:16Z

Simon: unlogged

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

Changing unlogged and logged tables in same transaction can result in apparently strange outcomes since the unlogged tables aren't replicated.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T15:41:23Z

Simon:

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns can result in "false conflicts", where there is conflict in terms of the data, just in terms of the row update. Such conflicts will result in just one of those changes being made, the other discarded according to last update wins.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-12T15:32:56Z

Simon: NOTIFY

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

NOTIFY is not supported across log based replication, either physical or logical.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master
* bdr.<nodename>.local_dbname = 'xxx' # optional parameter to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)
* bdr.synchronous_commit = ...; # optional parameter to set the synchronous_commit parameter the apply processes will be using
* max_logical_slots = X # set to the number of remotes

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns will result in just one of those changes being made, the other discarded.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T16:27:02Z

Simon: /* Very High Availability Multi-Master */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns will result in just one of those changes being made, the other discarded.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T16:26:29Z

Simon: /* Very High Availability Multi-Master */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns will result in just one of those changes being made, the other discarded.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T16:26:12Z

Simon:

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Bravo" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="bravo" # list of upstream master nodenames
** bdr.bravo.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Bravo
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Bravo" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Charlie" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Bravo using physical streaming with sync replication
** Server "Bravo" - Physical Standby - feeds changes to Charlie, Echo using logical streaming

* Site 2
** Server "Charlie" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Charlie using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns will result in just one of those changes being made, the other discarded.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T14:09:46Z

Simon: /* Data Conflicts */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

All conflicts are resolved at row level. Concurrent updates that touch completely separate columns will result in just one of those changes being made, the other discarded.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T14:08:44Z

Simon: /* Replication of DML changes */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

The values applied are the resulting values from the original UPDATE, including any modifications from before-row triggers, rules or functions. Any reflexive conditions, such as N = N+ 1 are resolved to their final value and volatile or stable functions carry their original values with them.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T14:04:28Z

Simon: /* Data Conflicts */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

Conflicts are logged if we specify bdr.log_conflicts = on

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T14:02:11Z

Simon: /* Data Conflicts */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution. It is important that these conflicts are resolved in an idempotent, similar manner so that all servers end with identical results.

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using system identifier from pg_control (currently).

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T13:54:21Z

Simon: /* Data Conflicts */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

Concurrent updates are resolved using last-update-wins strategy using timestamps. Should timestamps be identical, the tie is broken using node priority.

Updates and Inserts may cause uniqueness violation errors when changes are applied at remote nodes. These are not easily resolvable and represent severe application errors that cause the database contents of multiple servers to diverge from each other. Hence these are known as "divergent conflicts". Currently, replication stops should a divergent conflict occur.

Updates which cannot locate a row are presumed to be Delete/Update conflicts. These are counted but the Update is discarded.

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T13:48:34Z

Simon: /* Global Sequences */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. This is important with optimistic conflict resolution schemes because uniqueness violations are "divergent errors" and are not easily resolvable.

SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T13:46:29Z

Simon: /* Global Sequences */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Conflict Avoidance ==

=== Distributed Locking ===

Some clustering systems use distributed lock mechanisms to prevent concurrent access to data. These can perform reasonably when servers are very close but cannot support geographically distributed applications. Distributed locking is essentially a pessimistic approach, whereas BDR advocates an optimistic approach: avoid conflicts where possible though allow some types of conflict to occur and then resolve them when that happens.

=== Global Sequences ===

Many applications require unique values be assigned to database entries. Some applications use GUIDs generated by external programs, some use database-supplied values. SQL Standard requires Sequence objects which provide unique values, though these are isolated to a single node. These can then used to supply default values using DEFAULT nextval('mysequence'), as with the SERIAL datatype.

BDR requires Sequences to work together across multiple nodes. This is implemented as a new SequenceAccessMethod API (SeqAM), which allows plugins that provide get/set functions for sequences. Global Sequences are then implemented as a plugin which implements the SeqAM API and communicates across nodes to allow new ranges of values to be stored for each sequence.

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T13:31:28Z

Simon: /* Conflict Detection & Resolution */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

=== Examples ===

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T13:23:13Z

Simon: /* Lock Conflicts */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include
* explicit row level locking
* locking from foreign keys
* implicit locking because of row updates or deletes, from local activity or apply from other servers

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T13:18:26Z

Simon: /* Operational Issues and Debugging */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which are automatically re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include explicit row level locking, locking from foreign keys, implicit locking because of row updates or deletes.

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T09:29:57Z

Simon:

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLSR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLSR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which may be re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include explicit row level locking, locking from foreign keys, implicit locking because of row updates or deletes.

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T08:35:13Z

Simon: /* Data Conflicts */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLDR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLDR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which may be re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include explicit row level locking, locking from foreign keys, implicit locking because of row updates or deletes.

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row.

* We update a row on Customer table on NodeA. The change from NodeA is applied to NodeB just as we are inserting an activity on NodeB. The inserted activity causes a FK check....

[[Category:Replication]]

BDR User Guide

2013-03-08T08:24:38Z

Simon: /* Logical Log Streaming Replication */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to "streaming replication" i.e. physical log streaming replication (PLSR) from a user perspective - the main and big difference is that the receiving server is also a full master database that is non-readonly and can make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLDR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLDR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which may be re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include explicit row level locking, locking from foreign keys, implicit locking because of row updates or deletes.

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row. We update

[[Category:Replication]]

BDR User Guide

2013-03-08T08:23:13Z

Simon: /* New/Changed Parameter Reference */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to physical log streaming replication from a user perspective - the main and big difference is that the receiving server is also a full master database that can also make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLDR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLDR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master. Security for LLSR is identical to physical log streaming replication

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which may be re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include explicit row level locking, locking from foreign keys, implicit locking because of row updates or deletes.

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row. We update

[[Category:Replication]]

BDR User Guide

2013-03-08T08:21:32Z

Simon: /* User Guide */

BDR stands for '''B'''i'''D'''rectional '''R'''eplication.

Design work began in late 2011 to look at ways of adding new features to PostgreSQL core to support a flexible new infrastructure for replication that built upon and enhanced the existing streaming replication features added in 9.1-9.2. Initial design and project planning was by Simon Riggs; implementation lead is now Andres Freund, both from [http://www.2ndQuadrant.com/ 2ndQuadrant]. Various additional development contributions from the wider 2ndQuadrant team as well as reviews and input from other community devs.

At the [[PgCon2012CanadaInCoreReplicationMeeting]] an inital version of the design was presented. A presentation containing reasons leading to the current design and a prototype of it, including preliminary performance results, is [[:File:BDR_Presentation_PGCon2012.pdf|available here]].

= Project Overview and Plans =
== Project Aims ==
* in core
* fast
* reusable individual parts (see below), usable by other projects (slony, ...)
* basis for easier sharding/write scalability
* wide geographic distribution of replicated nodes

== High Level Planning ==
=== 9.3 ===
Fundamental changes have been made as part of 9.3 to support BDR; total of 16 separate commits on these and other smaller aspects

* background workers
* xlogreader implementation
* pg_xlogdump

Fully working implementation will be available for production use in 2013. At this stage, probably more than 50% of code exists out of core.

Exact mechanism for dissemination is not yet announced; key objective is to develop code with the objective of being core/contrib modules. There is no long term plan for existence of code outside of core.

=== 9.4 ===
Objective to implement main BDR features into core Postgres.

=== 9.5 ===
Additional features based upon feedback

== Aspects of BDR ==

Bi-Directional Replication consists of a number of related features

* Logical Log Streaming Replication - getting data from one master to another.
* Global Sequences - ability to support sequences that work globally across a set of nodes
* Conflict Detection & Resolution (options)
* DDL Replication via Event Triggers

Taken together these features will allow replication in both directions for any pair of servers. We could call this "multi-master replication", but the possibilities for constructing complex networks of servers allow much more than that, so we use the more general term bi-directional replication.

Note that these features aren't "clustering" in the sense that Oracle RAC uses the term. There is no distributed lock manager, global transaction coordinator etc.. The vision here is interconnected yet still separate servers, allowing each server to have radically different workloads and yet still work together, even across global scale and large geographic separation.

= User Guide =

== Logical Log Streaming Replication ==

Logical log streaming replication (LLSR) allows us to send changes from one master server to another master server. This is similar in many ways to physical log streaming replication from a user perspective - the main and big difference is that the receiving server is also a full master database that can also make changes. The master that sends data is also known as the upstream master and the master that receives data is also known as the downstream master. Data is sent in one direction only; setting up a configuration with data passing in both directions is called Bi-Directional Replication, discussed later.

The data that is replicated is change data in a special format that allows the changes to be logically reconstructed on the downstream master. The changes are generated by reading transaction log (WAL) data, making change capture on the upstream master much more efficient than trigger based replication, hence why we call this "logical log replication". Changes are passed from upstream to downstream using the libpq protocol, just as with physical log streaming replication.

One connection is required for each PostgreSQL database that is replicated. If two servers are connected, each of which has 50 databases then it would require 50 connections to send changes in one direction, from upstream to downstream. Each database connection must be specified, so it is possible to filter out unwanted databases simply by avoiding configuring replication for those databases.

Setting up replication for new databases is not (yet?) automatic, so additional configuration steps are required after CREATE DATABASE and this also requires a server restart. Adding replication for databases that do not exist yet will cause an ERROR, as will dropping a database that is being replicated.

Changes are handled by means of a BDR plugin, allowing multiple options. Current options are:

* pg_xlogdump - examines physical WAL records and produces textual debugging output (server program included in 9.3)
* textual output plugin - a demo plugin that generates SQL text (but doesn't apply changes)
* BDR apply process - applies logical changes to downstream master, making changes directly rather than generating SQL text and then parse/plan/executing SQL.

=== Replication of DML changes ===

All changes are replicated: INSERT, UPDATE, DELETE, TRUNCATE. Actions that generate WAL data but don't represent logical changes do not result in data transfer, e.g. full page writes, VACUUMs, hint bit setting. LLDR avoids much of the overhead from physical WAL, though does have message header overheads also, so bandwidth can be reduced in some, but not all cases.

(TRUNCATE currently not implemented yet)

LOCK statements are not replicated (possible future feature).

Temporary and Unlogged tables are not replicated. In contrast to physical standby servers, downstream masters can use temporary and unlogged tables.

DELETE and UPDATE statements that affect multiple rows on upstream master will cause a series of row changes on downstream master - these are likely to go at same speed as on origin, as long as an index is defined on the Primary Key of the table on the downstream master. INSERTs on upstream master do not require a unique constraint in order to replicate correctly. UPDATEs and DELETEs require some form of unique constraint, either PRIMARY KEY or UNIQUE NOT NULL.

UPDATEs that change the Primary Key of a table will be replicated correctly.

All columns are replicated on each table. Large column values that would be placed in TOAST tables are replicated without problem, avoiding de-compression and re-compression. If we update a row but do not change a TOASTed column value, then that data is not sent downstream.

All data types are handled, not just the built-in datatypes of PostgreSQL core. The only requirement is that user-defined types are installed identically in both upstream and downstream master.

Current plugin is binary only, requiring upstream and downstream master to use same CPU architecture and word-length, i.e. "identical servers", as with physical replication.

A textual output option will be available for passing data between non-identical servers, e.g. laptops communicating with a central server.

Changes are accumulated in memory (spilling to disk where required) and then sent to the downstream server at commit time. Aborted transactions are never sent. Application of changes on downstream master is currently single-threaded, though this process is effeciently implemented. Parallel apply is a possible future feature, especially for changes made while holding AccessExclusiveLock.

Changes are applied to the downstream master in commit sequence. This is a known-good serialization ordering of changes, so no replication failures are possible, as can happen with statement based replication (e.g. MySQL) or trigger based replication (e.g. Slony version 2.0). Users should note that this means the original order of locking of tables is not maintained. Although lock order is provably not an issue for the set of locks held on upstream master, additional locking on downstream side could cause lock waits or deadlocking in some cases. (Discussed in further detail later).

Larger transactions scroll to disk on the upstream master once they reach a certain size. Currently, large transactions can cause increased latency. Future enhancement will be to stream changes to downstream master once they fill the upstream memory buffer, though this is likely to be implemented in 9.5.

SET statements and parameter settings are not replicated. This has no effect on replication since we only replicate actual changes, not anything at SQL statement level. This means that we always update the correct tables, whatever the setting of search_path.

In some cases, additional deadlocks can occur on apply. This causes a retry of the apply of the replaying transaction.

Lock waits would cause latency problems/apply delays. This only applies to LOCK statements and DDL.

=== Table definitions and DDL replication ===

DML changes are replicated between tables with matching Schemaname.Tablename on both upstream and downstream masters. e.g. changes from Public.MyTable will go to Public.MyTable and MySchema.MyTable will go to MySchema.MyTable. This works even when no schema is specified on the original SQL since we identify the changed table from its internal OIDs in WAL records and then map that to whatever internal identifier is used on the downstream node.

This requires careful and exact synchronisation of table definitions on each node otherwise ERRORs will be generated. There are no plans to implement working replication between dissimilar table definitions.

In general, "exact match" is the best guide. Current details (subject to change) are
* Secondary indexes may differ between nodes
* Constraints must match for BDR.
* Storage parameters must match.
* Table-level parameters, e.g. fillfactor, autovacuum may differ
* Inheritance must be the same

Triggers and Rules are NOT executed by apply on downstream side, equivalent to an enforced setting of session_replication_role = origin.

Replication of DDL changes between nodes will be possible using event triggers, but is not yet integrated with LLSR.

=== Selective Replication (Table/Row-level filtering) ===

LLSR doesn't yet support selection of data at table or row level, only at database level. It is a design goal to be able to support this in the future.

=== Other Terminology ===

(Physical) Streaming replication talks about Master and Standby, so we could also talk about Master and Physical Standby, and then use Master and Logical Standby to describe LLDR. That terminology doesn't work when we consider that replication might be bi-directional, or at could be reconfigured that way in the future.

Similarly, the terms Origin, Provider and Subcriber only work with one Origin.

=== Configuration ===

Upstream master

* wal_level = 'logical'
* max_logical_slots = X
* max_wal_senders = Y # Y = max_logical_slots plus any physical streaming requirements

Downstream master

* shared_preload_libraries = 'bdr'

* bdr.connections="name_of_upstream_master" # list of upstream master nodenames
* bdr.<nodename>.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* (Also need a parameter like bdr.<nodename>.local_dbname = 'xxx' to cover the case where the databasename on upstream and downstream master differ. Not yet implemented)

wal_keep_segments should be set to a value that allows for some downtime of server/network.

==== New/Changed Parameter Reference ====

bdr.connections - list of nodes that this server will connect to. For each name listed here there must be one bdr.<nodename>.dsn entry

bdr.<nodename>.dsn - "data source name" - connection info for connecting to upstream master

max_logical_slots - LLSR uses persistent slots in memory which are reserved for each node at server start

wal_level - allows a new setting of 'logical' which produces mildly enhanced WAL contents to allow decoding of the WAL back into a logical change stream.

=== Tuning ===

As a result of the architecture there are few physical tuning parameters. That may grow as the implementation matures, but not significantly.

There are no parameters for tuning transfer latency.

The only likely tunable is the amount of memory used to accumulate changes before we send them downstream. Similar in many ways to setting of shared_buffers and should be increased on larger machines.

A variant of hot_standby_feedback could be implemented also, though would likely need renaming.

The CRC check while reading WAL is not useful in this context and there will likely be an option to skip that for logical decoding since it can be a CPU bottleneck.

=== Operational Issues and Debugging ===

In LLSR there are no user-level ERRORs that have special meaning. Any ERRORs generated are likely to be serious problems of some kind, apart from apply deadlocks, which may be re-tried.

=== Monitoring ===

Some new/changed views are available for monitoring activity

* pg_stat_replication
* pg_stat_logical_decoding
* pg_stat_logical_replication

Object statistics are updated normally on downstream side, which is essential to maintain autovacuum operating normally. If there are no local writes, these two views should show matching results (unless stats have been reset).
* pg_stat_user_tables
* pg_statio_user_tables

Since indexes are used to apply changes, the identifying indexes on downstream side may appear more heavily used with workloads that perform UPDATEs and DELETEs by non-identfying indexes.
* pg_stat_user_indexes
* pg_statio_user_indexes

== Bi-Directional Replication Use Cases ==

Bi-Directional Replication is designed to allow a very wide range of server connection topologies. The simplest to understand would be two servers each sending their changes to the other, which would be produced by making each server the downstream master of the other and so using two connections for each database.

Logical and physical streaming replication are designed to work side-by-side. This means that a master can be replicating using physical streaming replication to a local standby server, while at the same time replicating logical changes to a remote downstream master. Logical replication works alongside cascading replication also, so a physical standby can feed changes to a downstream master, allowing upstream master sending to physical standby sending to downstream master.

=== Simple multi-master pair ===

* "HA Cluster"
** Server "Alpha" - Master
** Server "Beta" - Master

===== Configuration =====

* Alpha
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="beta" # list of upstream master nodenames
** bdr.beta.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

* Beta
** wal_level = 'logical'
** max_logical_slots = 3
** max_wal_senders = 4 # Y = max_logical_slots plus any physical streaming requirements
** wal_keep_segments = 5000
** shared_preload_libraries = 'bdr'
** bdr.connections="alpha" # list of upstream master nodenames
** bdr.alpha.dsn = 'dbname=postgres' # connection string for connection from downstream to upstream master

=== HA and Logical Standby ===
Downstream masters allow users to create temporary tables, so they can be used as reporting servers.

* "HA Cluster"
** Server "Alpha" - Current Master
** Server "Beta" - Physical Standby - unused, apart from as failover target for Alpha - potentially specified in synchronous_standby_names
** Server "Gamma" - "Logical Standby" - downstream master

=== Very High Availability Multi-Master ===
A typical configuration for remote multi-master would then be:

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha using logical streaming

Bandwidth between Site 1 and Site 2 is minimised

=== 3-remote site simple Multi-Master ===

BDR supports "all to all" connections, so the latency for any change being applied on other masters is minimised. (Note that early designs of multi-master were arranged for circular replication, which has latency issues with larger numbers of nodes)

* Site 1
** Server "Alpha" - Master - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Alpha, Gamma using logical streaming

===== Configuration =====

Using node names that match port numbers, for clarity

* config for 5440:
**port = 5440
** bdr.connections='node_5441,node_5442'
** bdr.node_5441.dsn='port=5441 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5442.dsn='port=5442 dbname=postgres'

* config for 5440:
** port = 5441
** bdr.connections='node_5440,node_5442'
** bdr.node_5440.dsn='port=5440 dbname=postgres'
** bdr.node_5441.dsn='port=5441 dbname=postgres'

=== 3-remote site Max Availability Multi-Master ===

* Site 1
** Server "Alpha" - Master - feeds changes to Beta using physical streaming with sync replication
** Server "Beta" - Physical Standby - feeds changes to Gamma, Echo using logical streaming

* Site 2
** Server "Gamma" - Master - feeds changes to Delta using physical streaming with sync replication
** Server "Delta" - Physical Standby - feeds changes to Alpha, Echo using logical streaming

* Site 3
** Server "Echo" - Master - feeds changes to Foxtrot using physical streaming with sync replication
** Server "Foxtrot" - Physical Standby - feeds changes to Alpha, Gamma using logical streaming

Bandwidth and latency between sites is minimised.

=== N-site symmetric cluster replication ===

Symmetric cluster is where all masters are connected to each other.

N=19 has been tested and works fine.

N masters requires N-1 connections to other masters, so practical limits are <100 servers, or less if you have many separate databases.

The amount of work caused by each change is O(N), so there is a much lower practical limit based upon resource limits. A future option to limit to filter rows/tables for replication becomes essential with larger or more heavily updated databases, which is planned.

=== Complex/Assymetric Replication ===

Variety of options are possible.

== Global Sequences ==

MORE DOCS REQUIRED HERE

== Conflict Detection & Resolution ==

=== Lock Conflicts ===

Changes from the upstream master are applied on the downstream master by a single apply process. That process needs to RowExclusiveLock on the changing table and be able to write lock the changing tuple(s). Concurrent activity will prevent those changes from being immediately applied because of lock waits. Use the log_lock_waits facility to look for issues there.

By concurrent activity on a row, we include explicit row level locking, locking from foreign keys, implicit locking because of row updates or deletes.

=== Data Conflicts ===

Concurrent updates and deletes may also cause data-level conflicts to occur, which then require conflict resolution (see below).

As an example, lets say we have two tables Activity and Customer. There is a Foreign Key from Activity to Customer, constraining us to only record activity rows that have a matching customer row. We update

[[Category:Replication]]