https://wiki.postgresql.org/api.php?action=feedcontributions&user=Barwick&feedformat=atomPostgreSQL wiki - User contributions [en]2024-03-29T14:07:44ZUser contributionsMediaWiki 1.35.13https://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=37423Foreign data wrappers2022-12-28T11:02:29Z<p>Barwick: Update firebird_fdw info</p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/pgspider/jdbc_fdw github]<br />
|<br />
|[https://github.com/pgspider/jdbc_fdw/blob/main/README.md README]<br />
|More recent than the above, advertises write support.<br />
|-<br />
| [https://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| [https://gdal.org/drivers/vector/index.html GDAL/OGR]<br />
| Native<br />
| MIT<br />
| [https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
| yum.postgresql.org, apt.postgresql.org, and part of PostGIS windows bundle (application stackbuilder)<br />
| <br />
| Can access many kinds of data sources (Relational databases, spreadsheets, CSV files, web feature services, etc). Uses the [https://gdal.org/ GDAL library] which supports hundreds of formats to access the data. Exposes vector data as PostGIS geometry columns if you have PostGIS installed. Works great with both spatial and non-spatial data.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|<br />
|-<br />
| APIs (via [https://hub.steampipe.io/plugins Steampipe plugins])<br />
| Native<br />
| CLI and FDW extension: AGPL, Plugins: Apache 2.0<br />
| [https://github.com/turbot/steampipe CLI on GitHub], [https://github.com/turbot/steampipe-postgres-fdw FDW extension on GitHub]<br />
| [https://steampipe.io/downloads Steampipe downloads]<br />
| [https://steampipe.io/docs Steampipe docs]<br />
| [https://steampipe.io Steampipe] bundles Postgres with an FDW extension that supports a growing ecosystem of plugins. The plugins consume APIs, map them to tables, and enable queries within and across APIs.<br />
|}<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/postgres-fdw.html documentation]<br />
|<br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
|<br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|FDW for MySQL<br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|DB2<br />
|Native<br />
|<br />
|[https://github.com/wolfgangbrandl/db2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.firebirdsql.org/ Firebird]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|[https://github.com/ibarwick/firebird_fdw/blob/master/README.md README]<br />
|version [https://github.com/ibarwick/firebird_fdw/releases/tag/1.3.0 1.3.0] released (2022-12)<br />
|-<br />
|[https://www.sqlite.org/index.html SQLite]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pgspider/sqlite_fdw github]<br />
|[https://pgxn.org/dist/sqlite_fdw PGXN]<br />
|[https://github.com/pgspider/sqlite_fdw/blob/master/README.md README]<br />
|An FDW for SQLite3 (write support and several pushdown optimization)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[https://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[https://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|Native<br />
|Apache<br />
|[https://github.com/ildus/clickhouse_fdw Github]<br />
|<br />
|[https://github.com/ildus/clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|Native<br />
|<br />
|[https://github.com/Percona-Lab/clickhousedb_fdw Github]<br />
|<br />
|[https://github.com/Percona-Lab/clickhousedb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
| [https://github.com/griddb/griddb_nosql GridDB]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/griddb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/griddb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| InfluxDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/influxdb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/influxdb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| [https://kafka.apache.org/ Kafka]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/kafka_fdw GitHub]<br />
|<br />
| [https://github.com/adjust/kafka_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|[https://multicorn.org/ Multicorn]<br />
|GPLv3<br />
|[https://github.com/sim51/neo4j-fdw Github]<br />
|<br />
|[https://github.com/sim51/neo4j-fdw/blob/master/README.adoc README]<br />
|FWD for Neo4j and also add a Cypher function to Pg<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/slamdata/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://github.com/facebook/rocksdb RocksDB]<br />
| Native<br />
| Apache<br />
| [https://github.com/vidardb/PostgresForeignDataWrapper Github]<br />
|<br />
| [https://github.com/vidardb/PostgresForeignDataWrapper/blob/master/README.md README]<br />
| FDW for RocksDB<br />
|-<br />
| [https://en.wikipedia.org/wiki/SPARQL SPARQL]<br />
| [https://github.com/pgsql-io/multicorn2 Multicorn2]<br />
| PostgreSQL<br />
| [https://github.com/sjstoelting/sparql_fdwfdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to]<br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| CSV / gzipped<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
|<br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
|<br />
|-<br />
| Multi-File<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| Parquet<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/parquet_fdw GitHub]<br />
|<br />
|<br />
| Foreign data wrapper for reading Parquet files using libarrow/libparquet<br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| XML<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [https://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [https://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [https://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
|<br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
|<br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
|<br />
|}<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ICAL<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
|<br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
|<br />
|-<br />
| IMAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
|<br />
|-<br />
| RSS<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed.<br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|-<br />
| pgsql-http<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pramsey/pgsql-http GitHub]<br />
| Compile<br />
| <br />
| Allows to query any http resource using CURL libs. By Paul Ramsey<br />
<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| [https://www.openarchives.org/pmh/ OAI-PMH]<br />
| Native<br />
| MIT<br />
| [https://github.com/jimjonesbr/oai_fdw/ GitHub]<br />
|<br />
| [https://github.com/jimjonesbr/oai_fdw/blob/main/README.md README]<br />
| A PostgreSQL Foreign Data Wrapper to access OAI-PMH repositories (Open Archives Initiative Protocol for Metadata Harvesting). This wrapper supports the OAI-PMH 2.0 Protocol.<br />
|-<br />
| Database.com<br />
| [https://multicorn.org/ Multicorn]<br />
| BSD<br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Dun & Badstreet<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
|<br />
|<br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| DynamoDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Facebook<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
|<br />
| [https://github.com/hakanensari/frankfurter GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Keycloak<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/schne324/foreign-keycloak-wrapper GitHub]<br />
| [https://pgxn.org/dist/foreign-keycloak-wrapper/ PGXN]<br />
| [https://github.com/schne324/foreign-keycloak-wrapper/blob/master/README.md README]<br />
| Direct database integration with the [https://www.keycloak.org Keycloak] open-source Identity/Access Management solution.<br />
|-<br />
| Mailchimp<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
|<br />
|<br />
| Beta<br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| S3CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
|<br />
|<br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| ParquetS3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/parquet_s3_fdw GitHub]<br />
|<br />
|<br />
| Foreign data wrapper for reading Parquet files using libarrow/libparquet on Amazon S3 / Minio<br />
|-<br />
| Telegram<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/telegram_fdw GitHub]<br />
|<br />
|<br />
| telegram_fdw is a Telegram BOT implemented using the PostgreSQL foreign data wrapper interface.<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
|<br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
|<br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [https://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Open Weather Map<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/owmfdw GitHub]<br />
|<br />
|<br />
| A FDW for Open Weather Map (single city)<br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elasticsearch<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/matthewfranglen/postgres-elasticsearch-fdw GitHub]<br />
|<br />
|<br />
| Supports up to PG 13, ES 7.<br />
|-<br />
| Google BigQuery<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
|<br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
|<br />
|<br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://www.openscg.com/bigsql/hadoopfdw/ Bitbucket]<br />
|<br />
|<br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Hive<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
|<br />
|<br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://impala.apache.org/ Impala]<br />
| Native<br />
| BSD<br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://arrow.apache.org/ Apache Arrow]<br />
| Native<br />
| GPLv2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
|<br />
| A part of PG-Strom feature; as a columnar data source with support of SSD-to-GPU Direct SQL <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
|<br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
|<br />
|<br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Log files<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OpenStack / Telemetry<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OS Query<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
|<br />
|<br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
|<br />
|<br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
|<br />
|<br />
| A foreign datawrapper for querying system stats based on [https://libstatgrab.org/ statgrab]<br />
|-<br />
| Environment Variables<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/pgsql-tw/envfdw GitHub]<br />
|<br />
|<br />
| envFDW is a forign data wrapper for processing environment variables<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| faker_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/faker_fdw GitHub]<br />
|<br />
|<br />
| faker_fdw is a foreign data wrapper for PostgreSQL that generates fake data.<br />
|-<br />
| fdw_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
|<br />
|<br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PPG<br />
| Native<br />
|<br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
|<br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www2.meethue.com/en-us/philips-hue-benefits Phillips Hue Lighting Systems]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Random Number<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
|<br />
|<br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| Native<br />
| PostgreSQL<br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
|<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator<br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|-<br />
| VMware vSphere<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/vspherefdw GitHub]<br />
|<br />
|<br />
| A PostgreSQL FDW to query your VMware vSphere service<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
|<br />
|<br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
|<br />
|<br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Black Hole<br />
|<br />
|<br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
|<br />
|<br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [https://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Meson_for_patch_authors&diff=37358Meson for patch authors2022-12-08T10:26:17Z<p>Barwick: link to devel docs</p>
<hr />
<div>From PostgreSQL 16, the [[Meson]] build system will be supported. In most cases, patches submitted for inclusion in PostgreSQL will need to update the Meson configuration accordingly. This article (WIP) aims to provide an overview of necessary adjustments, particularly for anyone not yet familiar with Meson.<br />
<br />
== Meson documentation ==<br />
<br />
The PostgreSQL Wiki [[Meson]] article provides a general overview.<br />
<br />
PostgreSQL devel documentation: [https://www.postgresql.org/docs/devel/install-meson.html Building and Installation with Meson]<br />
<br />
=== Meson installation ===<br />
<br />
See [[Meson#Installing_Meson|here]] for general installation details.<br />
<br />
Note that Meson (and the ninja build tool) are somewhat bleeding edge, and the versions provided by some current distributions may not work correctly.<br />
<br />
== TAP tests ==<br />
<br />
If a patch adds any TAP tests, these must be included in the relevant "meson.build" file, which should contain a section like this:<br />
<br />
'tap': {<br />
'tests': [<br />
't/001_basic.pl',<br />
],<br />
},<br />
<br />
E.g. to add a second test, change this section to<br />
<br />
'tap': {<br />
'tests': [<br />
't/001_basic.pl',<br />
't/002_advanced.pl',<br />
],<br />
},<br />
<br />
To verify any new entries are correctly recognised, it should be sufficient to (from the PostgreSQL source directory) execute the following:<br />
<br />
meson setup ~/devel/postgresql-builddir<br />
cd ~/devel/postgresql-builddir<br />
meson test --list<br />
<br />
which will produce a list of all tests recognized by Meson.</div>Barwickhttps://wiki.postgresql.org/index.php?title=Meson_for_patch_authors&diff=37335Meson for patch authors2022-11-16T02:24:49Z<p>Barwick: Add basic notes about Meson for patch authors</p>
<hr />
<div>From PostgreSQL 16, the [[Meson]] build system will be supported. In most cases, patches submitted for inclusion in PostgreSQL will need to update the Meson configuration accordingly. This article (WIP) aims to provide an overview of necessary adjustments, particularly for anyone not yet familiar with Meson.<br />
<br />
== Meson documentation ==<br />
<br />
The PostgreSQL Wiki [[Meson]] article provides a general overview.<br />
<br />
At the time of writing (November 2022), Meson is not covered in the PostgreSQL documentation itself; a patch is in progress ([https://commitfest.postgresql.org/40/3953/ Documentation for building with meson]); an early version is rendered [https://api.cirrus-ci.com/v1/artifact/task/6084297781149696/html_docs/html_docs/install-meson.html here].<br />
<br />
=== Meson installation ===<br />
<br />
See [[Meson#Installing_Meson|here]] for general installation details.<br />
<br />
Note that Meson (and the ninja build tool) are somewhat bleeding edge, and the versions provided by some current distributions may not work correctly.<br />
<br />
== TAP tests ==<br />
<br />
If a patch adds any TAP tests, these must be included in the relevant "meson.build" file, which should contain a section like this:<br />
<br />
'tap': {<br />
'tests': [<br />
't/001_basic.pl',<br />
],<br />
},<br />
<br />
E.g. to add a second test, change this section to<br />
<br />
'tap': {<br />
'tests': [<br />
't/001_basic.pl',<br />
't/002_advanced.pl',<br />
],<br />
},<br />
<br />
To verify any new entries are correctly recognised, it should be sufficient to (from the PostgreSQL source directory) execute the following:<br />
<br />
meson setup ~/devel/postgresql-builddir<br />
cd ~/devel/postgresql-builddir<br />
meson test --list<br />
<br />
which will produce a list of all tests recognized by Meson.</div>Barwickhttps://wiki.postgresql.org/index.php?title=Streaming_Replication&diff=37324Streaming Replication2022-11-05T01:26:51Z<p>Barwick: Update documentation link</p>
<hr />
<div>'''Streaming Replication''' (SR) provides the capability to continuously ship and<br />
apply the [https://www.postgresql.org/docs/current/wal.html WAL XLOG] records to some number of standby servers in order to keep them current.<br />
<br />
This feature was added to PostgreSQL 9.0. The discussion below is a developer oriented one that contains some out of date information, particularly for PostgreSQL 10 and later. Users of this feature should refer to the current [https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION PostgreSQL Streaming Replication Documentation].<br />
<br />
= Developer and historical details on the project =<br />
SR was developed for inclusion in PostgreSQL 9.0 by NTT OSS Center. The lead developer is [mailto:masao.fujii@gmail.com Masao Fujii]. [http://www.pgcon.org/2008/schedule/events/76.en.html Synchronous Log Shipping Replication Presentation] introduces the early design of the feature.<br />
<br />
= Usage =<br />
== Users Overview ==<br />
* '''Log-shipping'''<br />
** XLOG records generated in the primary are periodically shipped to the standby via the network.<br />
** In the existing warm standby, only records in a filled file are shipped, what's referred to as file-based log-shipping. In SR, XLOG records in partially-filled XLOG file are shipped too, implementing record-based log-shipping. This means the window for data loss in SR is usually smaller than in warm standby, unless the warm standby was also configured for record-based shipping (which is complicated to setup).<br />
** The content of XLOG files written to the standby are exactly the same as those on the primary. XLOG files shipped can be used for a normal recovery and PITR.<br />
* '''Multiple standbys'''<br />
** More than one standby can establish a connection to the primary for SR. XLOG records are concurrently shipped to all these standbys. The delay/death of a standby does not harm log-shipping to other standbys.<br />
** The maximum number of standbys can be specified as a GUC variable.<br />
* '''Continuous recovery'''<br />
** The standby continuously replays XLOG records shipped without using pg_standby.<br />
** XLOG records shipped are replayed as soon as possible without waiting until XLOG file has been filled. The combination of [[Hot Standby]] and SR would make the latest data inserted into the primary visible in the standby almost immediately.<br />
** The standby periodically removes old XLOG files which are no longer needed for recovery, to prevent excessive disk usage.<br />
* '''Setup'''<br />
** The start of log-shipping does not interfere with any query processing on the primary.<br />
** The standby can be started in various conditions.<br />
*** If there are XLOG files in archive directory and restore_command is supplied, at first those files are replayed. Then the standby requests XLOG records following the last applied one to the primary. This prevents XLOG files already present in the standby from being shipped again. Similarly, XLOG files in pg_xlog are also replayed before starting log-shipping.<br />
*** If there is no XLOG files on the standby, the standby requests XLOG records following the starting XLOG location of recovery (the redo starting location).<br />
* '''Connection settings and authentication'''<br />
** A user can configure the same settings as a normal connection to a connection for SR (e.g., keepalive, pg_hba.conf).<br />
* '''Activation'''<br />
** The standby can keep waiting for activation as long as a user likes. This prevents the standby from being automatically brought up by failure of recovery or network outage.<br />
* '''Progress report'''<br />
** The primary and standby report the progress of log-shipping in PS display.<br />
* '''Graceful shutdown'''<br />
** When smart/fast shutdown is requested, the primary waits to exit until XLOG records have been sent to the standby, up to the shutdown checkpoint record.<br />
<br />
== Restrictions ==<br />
* '''Synchronous log-shipping'''<br />
** By default, SR supports operates in asynchronous manner, so the commit command might return a "success" to a client before the corresponding XLOG records are shipped to the standby. To enable synchronous replication, see [http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION Synchronous Replication]<br />
* '''Replication beyond timeline'''<br />
** A user has to get a fresh backup whenever making the old standby catch up.<br />
* '''Clustering'''<br />
** Postgres doesn't provide any clustering feature.<br />
<br />
== How to Use ==<br />
<br />
NB: there is overlap between this section and [[Binary Replication Tutorial]]<br />
<br />
* '''1.''' Install postgres in the primary and standby server as usual. This requires only ''configure'', ''make'' and ''make install''.<br />
* '''2.''' Create the initial database cluster in the primary server as usual, using ''initdb''.<br />
* '''3.''' Create an user named replication with REPLICATION privileges.<br />
$ CREATE ROLE replication WITH REPLICATION PASSWORD 'password' LOGIN;<br />
* '''4.''' Set up connections and authentication on the primary so that the standby server can successfully connect to the ''replication'' pseudo-database on the primary.<br />
<br />
$ $EDITOR postgresql.conf<br />
<br />
listen_addresses = '192.168.0.10'<br />
<br />
$ $EDITOR pg_hba.conf<br />
<br />
# The standby server must connect with a user that has replication privileges.<br />
# TYPE DATABASE USER ADDRESS METHOD<br />
host replication replication 192.168.0.20/32 md5<br />
<br />
* '''5.''' Set up the streaming replication related parameters on the primary server.<br />
$ $EDITOR postgresql.conf<br />
<br />
# To enable read-only queries on a standby server, wal_level must be set to<br />
# "hot_standby". But you can choose "archive" if you never connect to the<br />
# server in standby mode.<br />
wal_level = hot_standby<br />
<br />
# Set the maximum number of concurrent connections from the standby servers.<br />
max_wal_senders = 5<br />
<br />
# To prevent the primary server from removing the WAL segments required for<br />
# the standby server before shipping them, set the minimum number of segments<br />
# retained in the pg_xlog directory. At least wal_keep_segments should be<br />
# larger than the number of segments generated between the beginning of<br />
# online-backup and the startup of streaming replication. If you enable WAL<br />
# archiving to an archive directory accessible from the standby, this may<br />
# not be necessary.<br />
wal_keep_segments = 32<br />
<br />
# Enable WAL archiving on the primary to an archive directory accessible from<br />
# the standby. If wal_keep_segments is a high enough number to retain the WAL<br />
# segments required for the standby server, this is not necessary.<br />
archive_mode = on<br />
archive_command = 'cp %p /path_to/archive/%f'<br />
* '''6.''' Start postgres on the primary server.<br />
* '''7.''' Make a base backup by copying the primary server's data directory to the standby server.<br />
<br />
** '''7.1.''' Do it with pg_(start|stop)_backup and rsync on the primary<br />
<br />
$ psql -c "SELECT pg_start_backup('label', true)"<br />
$ rsync -ac ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid<br />
$ psql -c "SELECT pg_stop_backup()"<br />
<br />
** '''7.2.''' Do it with pg_basebackup on the standby<br />
<br />
In version 9.1+, pg_basebackup can do the dirty work of fetching the entire data directory of your PostgreSQL installation from the primary and placing it onto the standby server.<br />
<br />
The prerequisite is that you make sure the standby's data directory is empty.<br />
<br />
Make sure to remove any tablespace directories as well. You can find those directories with:<br />
<br />
$ psql -c '\db'<br />
<br />
If you keep your postgresql.conf and other config files in PGDATA, you need a backup of postgresql.conf, to restore after pg_basebackup.<br />
<br />
After you've cleared all the directories, you can use the following command to directly stream the data from the primary onto your standby server.<br />
Run it as the database superuser, typically 'postgres', to make sure the permissions are preserved (use su, sudo or whatever other tool to make sure you're not root).<br />
<br />
$ pg_basebackup -h 192.168.0.10 -D /srv/pgsql/standby -P -U replication --xlog-method=stream<br />
<br />
In version 9.3+, you can also add the -R option so it creates a minimal recovery command file for step 9 below.<br />
<br />
If you backed up postgresql.conf, now restore it.<br />
<br />
* '''8.''' Set up replication-related parameters, connections and authentication in the standby server like the primary, so that the standby might work as a primary after failover.<br />
* '''9.''' Enable read-only queries on the standby server. But if wal_level is ''archive'' on the primary, leave hot_standby unchanged (i.e., off).<br />
$ $EDITOR postgresql.conf<br />
<br />
hot_standby = on<br />
* '''10.''' Create a recovery command file in the standby server; the following parameters are required for streaming replication.<br />
$ $EDITOR recovery.conf<br />
# Note that recovery.conf must be in the $PGDATA directory, even if the<br />
# main postgresql.conf file is located elsewhere.<br />
<br />
# Specifies whether to start the server as a standby. In streaming replication,<br />
# this parameter must to be set to on.<br />
standby_mode = 'on'<br />
<br />
# Specifies a connection string which is used for the standby server to connect<br />
# with the primary.<br />
primary_conninfo = 'host=192.168.0.10 port=5432 user=replication password=password'<br />
<br />
# Specifies a trigger file whose presence should cause streaming replication to<br />
# end (i.e., failover).<br />
trigger_file = '/path_to/trigger'<br />
<br />
# Specifies a command to load archive segments from the WAL archive. If<br />
# wal_keep_segments is a high enough number to retain the WAL segments<br />
# required for the standby server, this may not be necessary. But<br />
# a large workload can cause segments to be recycled before the standby<br />
# is fully synchronized, requiring you to start again from a new base backup.<br />
restore_command = 'cp /path_to/archive/%f "%p"'<br />
* '''11.''' Start postgres in the standby server. It will start streaming replication.<br />
* '''12.''' You can calculate the replication lag by comparing the current WAL write location on the primary with the last WAL location received/replayed by the standby. They can be retrieved using ''pg_current_xlog_location'' on the primary and the ''pg_last_xlog_receive_location''/''pg_last_xlog_replay_location'' on the standby, respectively.<br />
$ psql -c "SELECT pg_current_xlog_location()" -h192.168.0.10 (primary host)<br />
pg_current_xlog_location <br />
--------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_receive_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_receive_location <br />
-------------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_replay_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_replay_location <br />
------------------------------<br />
0/2000000<br />
(1 row)<br />
* '''13.''' You can also check the progress of streaming replication by using ''ps'' command.<br />
# The displayed LSNs indicate the byte position that the standby server has<br />
# written up to in the xlogs.<br />
[primary] $ ps -ef | grep sender<br />
postgres 6879 6831 0 10:31 ? 00:00:00 postgres: wal sender process postgres 127.0.0.1(44663) streaming 0/2000000<br />
<br />
[standby] $ ps -ef | grep receiver<br />
postgres 6878 6872 1 10:31 ? 00:00:01 postgres: wal receiver process streaming 0/2000000<br />
* How to do failover<br />
** Create the trigger file in the standby after the primary fails.<br />
* How to stop the primary or the standby server<br />
** Shut down it as usual (''pg_ctl stop'').<br />
* How to restart streaming replication after failover<br />
** Repeat the operations from '''6th'''; making a fresh backup, some configurations and starting the original primary as the standby. The primary server doesn't need to be stopped during these operations.<br />
* How to restart streaming replication after the standby fails<br />
** Restart postgres in the standby server after eliminating the cause of failure.<br />
* How to disconnect the standby from the primary<br />
** Create the trigger file in the standby while the primary is running. Then the standby would be brought up.<br />
* How to re-synchronize the stand-alone standby after isolation<br />
** Shut down the standby as usual. And repeat the operations from '''6th'''.<br />
* If you have more than one standby, promoting one will break the other(s). Update their recovery.conf settings to point to the new master, set recovery_target_timeline to 'latest', scp/rsync the pg_xlog directory, and restart the standby.<br />
<br />
= Todo =<br />
== v9.0 ==<br />
<br />
Moved to [[PostgreSQL_9.0_Open_Items]]<br />
<br />
=== Committed ===<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01455.php Retrying from archive and some refactoring around Read/FetchRecord().] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00395.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg02601.php SR wrongly treats the WAL-boundary.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00396.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01715.php Adjust SR for some later changes about wal-skipping.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00399.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00024.php VACUUM FULL unexpectedly writes an XLOG UNLOGGED record.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00038.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01754.php Add a message type header.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00037.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01536.php Documentation: Add a new "Replication" chapter.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00115.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00350.php Failed assertion during recovery of partial WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00124.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00712.php A PANIC error might occur in the standby because of a partially-filled archived WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00137.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00330.php Improve the standby messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00140.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01672.php pq_getbyte_if_available() is not working because the win32 socket emulation layer simply wasn't designed to deal with non-blocking sockets.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00198.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01488.php Walsender might emit unfit messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00239.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01236.php Streaming replication on win32, still broken.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00270.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00992.php Create new section for recovery.conf.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00295.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01824.php Assertion failure in walreceiver.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00356.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01717.php Forbid a startup of walsender during recovery, and emit a suitable message? Or allow walsender to be started also during recovery?] - [http://archives.postgresql.org/message-id/20100316090955.9A5107541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01003.php How do we clean down the archive without using pg_standby?] - [http://archives.postgresql.org/message-id/20100318091718.BC14D7541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01510.php File-based log shipping without pg_standby doesn't replay the WAL files in pg_xlog.] - [http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php commit]<br />
<br />
== v9.1 ==<br />
=== Synchronization capability ===<br />
* Introduce the replication mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'' and ''fsync''.<br />
** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
** ''recv'' or ''fsync'' makes transaction commit wait for XLOG to be received or fsynced by the standby, respectively.<br />
** (''apply'' makes transaction commit wait for XLOG to be replayed by the standby. This mode will be supported in v9.2 or later)<br />
** The replication mode is specified in recovery.conf of the standby as well as other parameters for replication.<br />
*** The startup process reads the replication mode from recovery.conf and shares it to walreceiver via new shared-memory variable.<br />
*** Walreceiver also shares it to walsender by using the replication handshake message (existing protocol needs to be extended).<br />
** Based on the replication mode, walreceiver sends the reply meaning that replication is done up to the specified location to the primary.<br />
*** In async, walreceiver doesn't need to send any reply other than end-of-replication message.<br />
*** In recv or fsync, walreceiver sends the reply just after receiving or flushing XLOG, respectively.<br />
*** New message type for the reply needs to be defined. The reply is sent as CopyData message.<br />
** Walreceiver writes all the outstanding XLOG to disk before shutting down.<br />
** Walsender receives the reply from the standby, updates the location of the last record replicated, and announces completion of replication.<br />
*** New shared-memory variable to keep that location is required.<br />
** When processing the commit command, backend waits for XLOG to be replicated to only the standbys which are in the recv or fsync replication mode.<br />
*** Also smart shutdown waits for XLOG of shutdown checkpoint to be replicated.<br />
* Required optimization<br />
** Walsender should send outstanding XLOG without waiting wal_sender_delay.<br />
*** When processing the commit command, backend signals walsender to send outstanding XLOG immediately.<br />
** Backend should exit the wait loop as soon as the reply arrives at the primary.<br />
*** When receiving the reply, walsender signals backends to get up from the sleep and determine whether to exit the wait loop by checking the location of the last XLOG replicated.<br />
*** Only backends waiting for XLOG to be replicated up to the location contained in the reply are sent the signal.<br />
** Walsender waits for the signal from backends and the reply from the standby at the same time, by using select/poll.<br />
** Walsender reads XLOG from not only disk but also shared memory (wal buffers).<br />
** Walreceiver should flush XLOG file only when XLOG file is switched or the related page is flushed.<br />
*** When startup process or bgwriter flushes the buffer page, it checks whether the related XLOG has already been flushed via shared memory (location of the last XLOG flushed).<br />
*** It flushes the buffer page, if XLOG file has already been flushed.<br />
*** It signals walreceiver to flush XLOG file immediately and waits for the flush to complete, if XLOG file has not been flushed yet.<br />
** While the standby is catching up with the primary, those servers should ignore the replication mode and perform asynchronous replication.<br />
*** After those servers have almost gotten into synchronization, they perform replication based on the specified replication mode.<br />
*** New replication states like 'catching-up', 'sync', etc need to be defined, and the state machine for them is required on both servers.<br />
*** Current replication state can be monitored on both servers via SQL.<br />
* Required timeout<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG is replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
<br />
== Future release ==<br />
* '''Synchronization capability'''<br />
** Introduce the synchronization mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'', ''fsync'' and ''apply''.<br />
*** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
*** ''recv'', ''fsync'' and ''apply'' makes transaction commit wait for XLOG records to be received, fsynced and applied on the standby, respectively.<br />
** Change walsender to be able to read XLOG from not only the disk but also shared memory.<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG records are replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
* '''Monitoring'''<br />
** Provide the capability to check the progress and gap of streaming replication via one query. A collaboration of HS and SR is necessary to provide that capability on the standby side.<br />
** Provide the capability to check if the specified repliation is in progress via a query. Also more detailed status information might be necessary, e.g, the standby is catching up now, has already gotten into sync, and so on.<br />
** Change the stats collector to collect the statistics information about replication, e.g., average delay of replication time.<br />
** Develop the tool to calculate the latest XLOG position from XLOG files. This is necessary to check the gap of replication after the server fails.<br />
** Also develop the tool to extract the user-readable contents from XLOG files. This is necessary to see the contents of the gap, and manually restore them.<br />
* '''Easy to Use'''<br />
** Introduce the parameters like:<br />
*** replication_halt_timeout - replication will halt if no data has been sent for this much time.<br />
*** replication_halt_segments - replication will halt if number of WAL files in pg_xlog exceeds this threshold.<br />
*** These parameters allow us to avoid disk overflow.<br />
** Add new feature which transfers also base backup via the direct connection between the primary and the standby.<br />
** Add new hooks like walsender_hook and walreceiver_hook to cooperate with the add-on program for compression like pglesslog.<br />
** Provide a graceful termination of replication via a query on the primary. On the standby, a trigger file mechanism already provides that capability.<br />
** Support replication beyond timeline. The timeline history files need to be shipped from the primary to the standby.<br />
* '''Robustness'''<br />
** Support keepalive in libpq. This is useful for a client and the standby to detect a failure of the primary immediately.<br />
* '''Miscellaneous'''<br />
** Standalone walreceiver tool, which connects to the primary, continuously receives and writes XLOG records, independently from postgres server.<br />
** Cascade streaming replication. Allow walsender to send XLOG to another standby during recovery.<br />
** WAL archiving during recovery.<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Streaming_Replication&diff=37323Streaming Replication2022-11-05T01:26:19Z<p>Barwick: Emphasize that this document is quite out of date; remove link to incomplete 9.0-only tutorial</p>
<hr />
<div>'''Streaming Replication''' (SR) provides the capability to continuously ship and<br />
apply the [http://www.postgresql.org/docs/current/static/wal.html WAL XLOG] records to some number of standby servers in order to keep them current.<br />
<br />
This feature was added to PostgreSQL 9.0. The discussion below is a developer oriented one that contains some out of date information, particularly for PostgreSQL 10 and later. Users of this feature should refer to the current [https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION PostgreSQL Streaming Replication Documentation].<br />
<br />
= Developer and historical details on the project =<br />
SR was developed for inclusion in PostgreSQL 9.0 by NTT OSS Center. The lead developer is [mailto:masao.fujii@gmail.com Masao Fujii]. [http://www.pgcon.org/2008/schedule/events/76.en.html Synchronous Log Shipping Replication Presentation] introduces the early design of the feature.<br />
<br />
= Usage =<br />
== Users Overview ==<br />
* '''Log-shipping'''<br />
** XLOG records generated in the primary are periodically shipped to the standby via the network.<br />
** In the existing warm standby, only records in a filled file are shipped, what's referred to as file-based log-shipping. In SR, XLOG records in partially-filled XLOG file are shipped too, implementing record-based log-shipping. This means the window for data loss in SR is usually smaller than in warm standby, unless the warm standby was also configured for record-based shipping (which is complicated to setup).<br />
** The content of XLOG files written to the standby are exactly the same as those on the primary. XLOG files shipped can be used for a normal recovery and PITR.<br />
* '''Multiple standbys'''<br />
** More than one standby can establish a connection to the primary for SR. XLOG records are concurrently shipped to all these standbys. The delay/death of a standby does not harm log-shipping to other standbys.<br />
** The maximum number of standbys can be specified as a GUC variable.<br />
* '''Continuous recovery'''<br />
** The standby continuously replays XLOG records shipped without using pg_standby.<br />
** XLOG records shipped are replayed as soon as possible without waiting until XLOG file has been filled. The combination of [[Hot Standby]] and SR would make the latest data inserted into the primary visible in the standby almost immediately.<br />
** The standby periodically removes old XLOG files which are no longer needed for recovery, to prevent excessive disk usage.<br />
* '''Setup'''<br />
** The start of log-shipping does not interfere with any query processing on the primary.<br />
** The standby can be started in various conditions.<br />
*** If there are XLOG files in archive directory and restore_command is supplied, at first those files are replayed. Then the standby requests XLOG records following the last applied one to the primary. This prevents XLOG files already present in the standby from being shipped again. Similarly, XLOG files in pg_xlog are also replayed before starting log-shipping.<br />
*** If there is no XLOG files on the standby, the standby requests XLOG records following the starting XLOG location of recovery (the redo starting location).<br />
* '''Connection settings and authentication'''<br />
** A user can configure the same settings as a normal connection to a connection for SR (e.g., keepalive, pg_hba.conf).<br />
* '''Activation'''<br />
** The standby can keep waiting for activation as long as a user likes. This prevents the standby from being automatically brought up by failure of recovery or network outage.<br />
* '''Progress report'''<br />
** The primary and standby report the progress of log-shipping in PS display.<br />
* '''Graceful shutdown'''<br />
** When smart/fast shutdown is requested, the primary waits to exit until XLOG records have been sent to the standby, up to the shutdown checkpoint record.<br />
<br />
== Restrictions ==<br />
* '''Synchronous log-shipping'''<br />
** By default, SR supports operates in asynchronous manner, so the commit command might return a "success" to a client before the corresponding XLOG records are shipped to the standby. To enable synchronous replication, see [http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION Synchronous Replication]<br />
* '''Replication beyond timeline'''<br />
** A user has to get a fresh backup whenever making the old standby catch up.<br />
* '''Clustering'''<br />
** Postgres doesn't provide any clustering feature.<br />
<br />
== How to Use ==<br />
<br />
NB: there is overlap between this section and [[Binary Replication Tutorial]]<br />
<br />
* '''1.''' Install postgres in the primary and standby server as usual. This requires only ''configure'', ''make'' and ''make install''.<br />
* '''2.''' Create the initial database cluster in the primary server as usual, using ''initdb''.<br />
* '''3.''' Create an user named replication with REPLICATION privileges.<br />
$ CREATE ROLE replication WITH REPLICATION PASSWORD 'password' LOGIN;<br />
* '''4.''' Set up connections and authentication on the primary so that the standby server can successfully connect to the ''replication'' pseudo-database on the primary.<br />
<br />
$ $EDITOR postgresql.conf<br />
<br />
listen_addresses = '192.168.0.10'<br />
<br />
$ $EDITOR pg_hba.conf<br />
<br />
# The standby server must connect with a user that has replication privileges.<br />
# TYPE DATABASE USER ADDRESS METHOD<br />
host replication replication 192.168.0.20/32 md5<br />
<br />
* '''5.''' Set up the streaming replication related parameters on the primary server.<br />
$ $EDITOR postgresql.conf<br />
<br />
# To enable read-only queries on a standby server, wal_level must be set to<br />
# "hot_standby". But you can choose "archive" if you never connect to the<br />
# server in standby mode.<br />
wal_level = hot_standby<br />
<br />
# Set the maximum number of concurrent connections from the standby servers.<br />
max_wal_senders = 5<br />
<br />
# To prevent the primary server from removing the WAL segments required for<br />
# the standby server before shipping them, set the minimum number of segments<br />
# retained in the pg_xlog directory. At least wal_keep_segments should be<br />
# larger than the number of segments generated between the beginning of<br />
# online-backup and the startup of streaming replication. If you enable WAL<br />
# archiving to an archive directory accessible from the standby, this may<br />
# not be necessary.<br />
wal_keep_segments = 32<br />
<br />
# Enable WAL archiving on the primary to an archive directory accessible from<br />
# the standby. If wal_keep_segments is a high enough number to retain the WAL<br />
# segments required for the standby server, this is not necessary.<br />
archive_mode = on<br />
archive_command = 'cp %p /path_to/archive/%f'<br />
* '''6.''' Start postgres on the primary server.<br />
* '''7.''' Make a base backup by copying the primary server's data directory to the standby server.<br />
<br />
** '''7.1.''' Do it with pg_(start|stop)_backup and rsync on the primary<br />
<br />
$ psql -c "SELECT pg_start_backup('label', true)"<br />
$ rsync -ac ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid<br />
$ psql -c "SELECT pg_stop_backup()"<br />
<br />
** '''7.2.''' Do it with pg_basebackup on the standby<br />
<br />
In version 9.1+, pg_basebackup can do the dirty work of fetching the entire data directory of your PostgreSQL installation from the primary and placing it onto the standby server.<br />
<br />
The prerequisite is that you make sure the standby's data directory is empty.<br />
<br />
Make sure to remove any tablespace directories as well. You can find those directories with:<br />
<br />
$ psql -c '\db'<br />
<br />
If you keep your postgresql.conf and other config files in PGDATA, you need a backup of postgresql.conf, to restore after pg_basebackup.<br />
<br />
After you've cleared all the directories, you can use the following command to directly stream the data from the primary onto your standby server.<br />
Run it as the database superuser, typically 'postgres', to make sure the permissions are preserved (use su, sudo or whatever other tool to make sure you're not root).<br />
<br />
$ pg_basebackup -h 192.168.0.10 -D /srv/pgsql/standby -P -U replication --xlog-method=stream<br />
<br />
In version 9.3+, you can also add the -R option so it creates a minimal recovery command file for step 9 below.<br />
<br />
If you backed up postgresql.conf, now restore it.<br />
<br />
* '''8.''' Set up replication-related parameters, connections and authentication in the standby server like the primary, so that the standby might work as a primary after failover.<br />
* '''9.''' Enable read-only queries on the standby server. But if wal_level is ''archive'' on the primary, leave hot_standby unchanged (i.e., off).<br />
$ $EDITOR postgresql.conf<br />
<br />
hot_standby = on<br />
* '''10.''' Create a recovery command file in the standby server; the following parameters are required for streaming replication.<br />
$ $EDITOR recovery.conf<br />
# Note that recovery.conf must be in the $PGDATA directory, even if the<br />
# main postgresql.conf file is located elsewhere.<br />
<br />
# Specifies whether to start the server as a standby. In streaming replication,<br />
# this parameter must to be set to on.<br />
standby_mode = 'on'<br />
<br />
# Specifies a connection string which is used for the standby server to connect<br />
# with the primary.<br />
primary_conninfo = 'host=192.168.0.10 port=5432 user=replication password=password'<br />
<br />
# Specifies a trigger file whose presence should cause streaming replication to<br />
# end (i.e., failover).<br />
trigger_file = '/path_to/trigger'<br />
<br />
# Specifies a command to load archive segments from the WAL archive. If<br />
# wal_keep_segments is a high enough number to retain the WAL segments<br />
# required for the standby server, this may not be necessary. But<br />
# a large workload can cause segments to be recycled before the standby<br />
# is fully synchronized, requiring you to start again from a new base backup.<br />
restore_command = 'cp /path_to/archive/%f "%p"'<br />
* '''11.''' Start postgres in the standby server. It will start streaming replication.<br />
* '''12.''' You can calculate the replication lag by comparing the current WAL write location on the primary with the last WAL location received/replayed by the standby. They can be retrieved using ''pg_current_xlog_location'' on the primary and the ''pg_last_xlog_receive_location''/''pg_last_xlog_replay_location'' on the standby, respectively.<br />
$ psql -c "SELECT pg_current_xlog_location()" -h192.168.0.10 (primary host)<br />
pg_current_xlog_location <br />
--------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_receive_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_receive_location <br />
-------------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_replay_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_replay_location <br />
------------------------------<br />
0/2000000<br />
(1 row)<br />
* '''13.''' You can also check the progress of streaming replication by using ''ps'' command.<br />
# The displayed LSNs indicate the byte position that the standby server has<br />
# written up to in the xlogs.<br />
[primary] $ ps -ef | grep sender<br />
postgres 6879 6831 0 10:31 ? 00:00:00 postgres: wal sender process postgres 127.0.0.1(44663) streaming 0/2000000<br />
<br />
[standby] $ ps -ef | grep receiver<br />
postgres 6878 6872 1 10:31 ? 00:00:01 postgres: wal receiver process streaming 0/2000000<br />
* How to do failover<br />
** Create the trigger file in the standby after the primary fails.<br />
* How to stop the primary or the standby server<br />
** Shut down it as usual (''pg_ctl stop'').<br />
* How to restart streaming replication after failover<br />
** Repeat the operations from '''6th'''; making a fresh backup, some configurations and starting the original primary as the standby. The primary server doesn't need to be stopped during these operations.<br />
* How to restart streaming replication after the standby fails<br />
** Restart postgres in the standby server after eliminating the cause of failure.<br />
* How to disconnect the standby from the primary<br />
** Create the trigger file in the standby while the primary is running. Then the standby would be brought up.<br />
* How to re-synchronize the stand-alone standby after isolation<br />
** Shut down the standby as usual. And repeat the operations from '''6th'''.<br />
* If you have more than one standby, promoting one will break the other(s). Update their recovery.conf settings to point to the new master, set recovery_target_timeline to 'latest', scp/rsync the pg_xlog directory, and restart the standby.<br />
<br />
= Todo =<br />
== v9.0 ==<br />
<br />
Moved to [[PostgreSQL_9.0_Open_Items]]<br />
<br />
=== Committed ===<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01455.php Retrying from archive and some refactoring around Read/FetchRecord().] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00395.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg02601.php SR wrongly treats the WAL-boundary.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00396.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01715.php Adjust SR for some later changes about wal-skipping.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00399.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00024.php VACUUM FULL unexpectedly writes an XLOG UNLOGGED record.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00038.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01754.php Add a message type header.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00037.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01536.php Documentation: Add a new "Replication" chapter.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00115.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00350.php Failed assertion during recovery of partial WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00124.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00712.php A PANIC error might occur in the standby because of a partially-filled archived WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00137.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00330.php Improve the standby messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00140.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01672.php pq_getbyte_if_available() is not working because the win32 socket emulation layer simply wasn't designed to deal with non-blocking sockets.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00198.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01488.php Walsender might emit unfit messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00239.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01236.php Streaming replication on win32, still broken.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00270.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00992.php Create new section for recovery.conf.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00295.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01824.php Assertion failure in walreceiver.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00356.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01717.php Forbid a startup of walsender during recovery, and emit a suitable message? Or allow walsender to be started also during recovery?] - [http://archives.postgresql.org/message-id/20100316090955.9A5107541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01003.php How do we clean down the archive without using pg_standby?] - [http://archives.postgresql.org/message-id/20100318091718.BC14D7541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01510.php File-based log shipping without pg_standby doesn't replay the WAL files in pg_xlog.] - [http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php commit]<br />
<br />
== v9.1 ==<br />
=== Synchronization capability ===<br />
* Introduce the replication mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'' and ''fsync''.<br />
** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
** ''recv'' or ''fsync'' makes transaction commit wait for XLOG to be received or fsynced by the standby, respectively.<br />
** (''apply'' makes transaction commit wait for XLOG to be replayed by the standby. This mode will be supported in v9.2 or later)<br />
** The replication mode is specified in recovery.conf of the standby as well as other parameters for replication.<br />
*** The startup process reads the replication mode from recovery.conf and shares it to walreceiver via new shared-memory variable.<br />
*** Walreceiver also shares it to walsender by using the replication handshake message (existing protocol needs to be extended).<br />
** Based on the replication mode, walreceiver sends the reply meaning that replication is done up to the specified location to the primary.<br />
*** In async, walreceiver doesn't need to send any reply other than end-of-replication message.<br />
*** In recv or fsync, walreceiver sends the reply just after receiving or flushing XLOG, respectively.<br />
*** New message type for the reply needs to be defined. The reply is sent as CopyData message.<br />
** Walreceiver writes all the outstanding XLOG to disk before shutting down.<br />
** Walsender receives the reply from the standby, updates the location of the last record replicated, and announces completion of replication.<br />
*** New shared-memory variable to keep that location is required.<br />
** When processing the commit command, backend waits for XLOG to be replicated to only the standbys which are in the recv or fsync replication mode.<br />
*** Also smart shutdown waits for XLOG of shutdown checkpoint to be replicated.<br />
* Required optimization<br />
** Walsender should send outstanding XLOG without waiting wal_sender_delay.<br />
*** When processing the commit command, backend signals walsender to send outstanding XLOG immediately.<br />
** Backend should exit the wait loop as soon as the reply arrives at the primary.<br />
*** When receiving the reply, walsender signals backends to get up from the sleep and determine whether to exit the wait loop by checking the location of the last XLOG replicated.<br />
*** Only backends waiting for XLOG to be replicated up to the location contained in the reply are sent the signal.<br />
** Walsender waits for the signal from backends and the reply from the standby at the same time, by using select/poll.<br />
** Walsender reads XLOG from not only disk but also shared memory (wal buffers).<br />
** Walreceiver should flush XLOG file only when XLOG file is switched or the related page is flushed.<br />
*** When startup process or bgwriter flushes the buffer page, it checks whether the related XLOG has already been flushed via shared memory (location of the last XLOG flushed).<br />
*** It flushes the buffer page, if XLOG file has already been flushed.<br />
*** It signals walreceiver to flush XLOG file immediately and waits for the flush to complete, if XLOG file has not been flushed yet.<br />
** While the standby is catching up with the primary, those servers should ignore the replication mode and perform asynchronous replication.<br />
*** After those servers have almost gotten into synchronization, they perform replication based on the specified replication mode.<br />
*** New replication states like 'catching-up', 'sync', etc need to be defined, and the state machine for them is required on both servers.<br />
*** Current replication state can be monitored on both servers via SQL.<br />
* Required timeout<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG is replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
<br />
== Future release ==<br />
* '''Synchronization capability'''<br />
** Introduce the synchronization mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'', ''fsync'' and ''apply''.<br />
*** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
*** ''recv'', ''fsync'' and ''apply'' makes transaction commit wait for XLOG records to be received, fsynced and applied on the standby, respectively.<br />
** Change walsender to be able to read XLOG from not only the disk but also shared memory.<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG records are replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
* '''Monitoring'''<br />
** Provide the capability to check the progress and gap of streaming replication via one query. A collaboration of HS and SR is necessary to provide that capability on the standby side.<br />
** Provide the capability to check if the specified repliation is in progress via a query. Also more detailed status information might be necessary, e.g, the standby is catching up now, has already gotten into sync, and so on.<br />
** Change the stats collector to collect the statistics information about replication, e.g., average delay of replication time.<br />
** Develop the tool to calculate the latest XLOG position from XLOG files. This is necessary to check the gap of replication after the server fails.<br />
** Also develop the tool to extract the user-readable contents from XLOG files. This is necessary to see the contents of the gap, and manually restore them.<br />
* '''Easy to Use'''<br />
** Introduce the parameters like:<br />
*** replication_halt_timeout - replication will halt if no data has been sent for this much time.<br />
*** replication_halt_segments - replication will halt if number of WAL files in pg_xlog exceeds this threshold.<br />
*** These parameters allow us to avoid disk overflow.<br />
** Add new feature which transfers also base backup via the direct connection between the primary and the standby.<br />
** Add new hooks like walsender_hook and walreceiver_hook to cooperate with the add-on program for compression like pglesslog.<br />
** Provide a graceful termination of replication via a query on the primary. On the standby, a trigger file mechanism already provides that capability.<br />
** Support replication beyond timeline. The timeline history files need to be shipped from the primary to the standby.<br />
* '''Robustness'''<br />
** Support keepalive in libpq. This is useful for a client and the standby to detect a failure of the primary immediately.<br />
* '''Miscellaneous'''<br />
** Standalone walreceiver tool, which connects to the primary, continuously receives and writes XLOG records, independently from postgres server.<br />
** Cascade streaming replication. Allow walsender to send XLOG to another standby during recovery.<br />
** WAL archiving during recovery.<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Binary_Replication_Tutorial&diff=37322Binary Replication Tutorial2022-11-05T01:22:58Z<p>Barwick: Note this is very outdated; update master/slave terminology</p>
<hr />
<div>Welcome to the new PostgreSQL 9 replication and standby databases guide. This new set of features implements possibly the longest awaited functionality in PostgreSQL's history. As a result, a lot of people are going to be trying to deploy standby databases for the first time, and find the process rather unintuitive. This guide is here to help.<br />
<br />
'''This tutorial is incomplete and outdated'''<br />
<br />
NB: there is some duplication with the page on [[Streaming Replication]]<br />
<br />
= 5 Minutes to Simple Replication =<br />
<br />
This is the easiest way to set up replication between a primary and standby. It requires shutting down the primary; other methods are detailed later in this guide.<br />
<br />
What we're going to do is shut down the primary and copy the files we need over to the standby server, creating a cloned copy of the primary. Because the primary is shut down, we don't have to worry about changes being made to it.<br />
<br />
Note: Both the '5 minutes' instructions and the '10 minutes' version which follows do not deal with the complications that arise with a database that uses tablespaces, specifically what to do about the pg_tblspc directory and its contents.<br />
<br />
== Prerequisites ==<br />
<br />
You must have the right setup to make this work:<br />
<br />
* 2 servers with similar operating systems (e.g both Linux 64-bit).<br />
* The same release of PostgreSQL 9.0 installed on both servers.<br />
* PostgreSQL superuser shell access on both servers.<br />
* Knowledge of how to start, stop and reload Postgres.<br />
* PostgreSQL 9.0 running on Server1.<br />
* A database created and loaded on Server1.<br />
* A postgres user or root user who has network <br />
<br />
See the full documentation for more information:<br />
<br />
* [http://www.postgresql.org/docs/9.0/static/warm-standby.html 9.0 Replication Documentation]<br />
* [http://www.postgresql.org/docs/9.1/static/warm-standby.html 9.1 Replication Documentation]<br />
<br />
== Binary Replication in 7 Steps ==<br />
<br />
This 6-step guide, and all of the examples in this tutorial, assume that you have a primary server at 192.168.0.1 and a standby server at 192.168.0.2 and that your database and its configuration files are installed at /var/lib/postgresql/data. Replace those with whatever your actual server addresses and directories are.<br />
<br />
1. Edit postgresql.conf on the primary to turn on streaming replication. Change these settings:<br />
<br />
listen_addresses = '*'<br />
wal_level = hot_standby<br />
max_wal_senders = 3<br />
<br />
2. Edit pg_hba.conf on the primary in order to let the standby connect. <br />
<br />
host replication all 192.168.0.2/32 trust<br />
<br />
3. Edit postgresql.conf on the standby to set up hot standby. Change this line:<br />
<br />
hot_standby = on<br />
<br />
4. Create or edit recovery.conf on the standby to set up replication and standby mode. Save a file in the standby's '''data directory''', called recovery.conf, with the following lines:<br />
<br />
standby_mode = 'on'<br />
primary_conninfo = 'host=192.168.0.1'<br />
<br />
5. Shut down both the primary and standby, and copy the files. You want to copy most but not all files between the two servers, excluding the configuration files and the pg_xlog directory. An example rsync script would be:<br />
<br />
rsync -av --exclude pg_xlog --exclude postgresql.conf data/* 192.168.0.2:/var/lib/postgresql/data/<br />
<br />
6. Start the standby first, so that they can't get out of sync. (Messages will be logged about not being able to connect to the primary server, that's OK.)<br />
<br />
7. Start the primary.<br />
<br />
== Starting Replication with only a Quick Primary Restart ==<br />
<br />
Is taking down the primary for long enough to copy the files too long? Then you need the 10-minute version.<br />
<br />
What we're going to do this time is similar to what we did before, cloning the database by copying the files from the primary to the standby server. However, because the database is only going to be shut down for a short period of time, long enough to activate the changes in the configuration file, after we've copied the data files we will need to copy additional files so that the standby will be an up-to-date copy of the primary. <br />
<br />
So, we will tell the primary we're running a backup, copy the data files (not quite the same set of files as before), tell the primary the backup is complete, then copy the WAL files in the pg_xlog directory so that when the standby comes up it can make all the changes that were committed to the primary database after the backup was started.<br />
<br />
First, start with the same prerequisites as above.<br />
<br />
1. Set the postgresql.conf variables the same in step (1) as above.<br />
<br />
2. Don't close the file yet. You'll need to set two other variables which control the size of your write-ahead-log (WAL). The first is wal_keep_segments, the second is checkpoint_segments. Unless you've already done so, you're going to need to increase these, which is usually a good idea for performance anyway. You want the WAL to be big enough to not get used up in 15 or 20 minutes. If you don't have a clear idea of that, here's some reasonable values, based on how busy and how large your database is. Also, a database with large blob objects may require a much larger setting. Remember, these logs will take up disk space, so make sure that you have enough available - space requirements are below.<br />
<br />
checkpoint_segments = 8 <br />
wal_keep_segments = 8 <br />
# light load 500MB<br />
<br />
checkpoint_segments = 16<br />
wal_keep_segments = 32<br />
# moderately busy 1.5GB <br />
<br />
checkpoint_segments = 64<br />
wal_keep_segments = 128<br />
# busy server 5GB<br />
<br />
You don't ''have'' to increase checkpoint_segments in order to increase wal_keep_segments, but it's generally a good idea. Now save the file. <br />
<br />
3. Edit pg_hba.conf as in (2) in the "Six Steps" above.<br />
<br />
4. Now you need to restart the primary. Given the interruption in service, you should probably plan this ahead. <br />
<br />
5. Edit postgresql.conf and recovery.conf on the standby as in (3) above.<br />
<br />
6. Now, we're going to need to copy the files from the primary and start the standby. Unlike in the 6-step version, this needs to be done quickly or the standby will fail to sync and you'll need to try again. First step, you need to tell the primary you're starting a backup (see below for a more detailed explanation of this). Log in to psql as the database superuser.<br />
<br />
psql -U postgres<br />
# select pg_start_backup('clone',true);<br />
<br />
Note that the string you use as a backup label doesn't matter; use any string you want.<br />
<br />
7. Now, quickly copy all the database files. This rsync is slightly different from the 6-step version:<br />
<br />
rsync -av --exclude pg_xlog --exclude postgresql.conf --exclude postgresql.pid \ <br />
data/* 192.168.0.2:/var/lib/postgresql/data/<br />
<br />
8. As soon as that's done you need to stop the backup on the primary:<br />
<br />
# select pg_stop_backup();<br />
<br />
9. As soon as that completes, you need to quickly copy the WAL files from the primary to the standby.<br />
<br />
rsync -av data/pg_xlog 192.168.0.2:/var/lib/postgresql/data/<br />
<br />
10. Now, start the standby.<br />
<br />
If you've done this quickly enough, then the standby should catch up with the primary and you should be replicating. If not, you'll get this message:<br />
<br />
(Future Revisions note: Message needs to go here)<br />
<br />
... which means you need to try again, possibly with checkpoint_segments and wal_keep_segments higher. If that still doesn't work, you're going to need to use the even more complex archiving method described below.<br />
<br />
Now, the rest of the guide will explain how to deal with more complex situations, such as archive logs, handling security, and maintaining availability, failover and standby promotion.<br />
<br />
= Introduction to Binary Replication =<br />
<br />
Binary replication is also called "Hot Standby" and "Streaming Replication" which are two separate, but complimentary, features of PostgreSQL 9.0 and later. Here's some general information about how they work and what they are for.<br />
<br />
== What Can You Do With Binary Replication? ==<br />
<br />
* Have a simple and complete replica of your production database, preventing all but a couple seconds of data loss even under catastrophic circumstances.<br />
* Load-balance between your read/write primary server and multiple read-only standby servers. (Note: This means that queries that aren't read-only cannot be run on a standby server. A common misconception has to do with finding the 'current' value of a sequence on a standby server, that is not possible.)<br />
* Run reporting or other long-running queries on a replica server, taking them off your main transaction-processing server.<br />
* Replicate all DDL, including table and index changes, and even creating new databases.<br />
* Replicate a hosted multi-tenant database, making no specific requirements for primary keys or database changes of your users.<br />
<br />
== What Can't You Do With Binary Replication? ==<br />
<br />
* Replicate a specific table, schema, or database. Binary replication is the entire Postgres instance (or "cluster").<br />
* Multi-primary replication. Multi-primary binary replication is probably technically impossible.<br />
* Replicate between different versions of PostgreSQL, or between different platforms.<br />
* Set up replication without administration rights on the server. Sorry, working on it.<br />
* Replicate data synchronously, guaranteeing zero data loss. And ... this is here since the release of PostgreSQL 9.1!<br />
<br />
For the reasons above, we expect that Slony-I, Londiste, Bucardo, pgPool2 and other systems will continue to be used.<br />
<br />
== Transaction Logs and Log Shipping ==<br />
<br />
Users who are already familiar with the PostgreSQL transaction log and warm standby can skip this section.<br />
<br />
An individual "instance", "server", or (confusingly) "cluster" of PostgreSQL (hereafter Server) consists of a single postprimary server process connected to a single initialized PostgreSQL data directory (PGDATA), which in turn contains several databases. Each running Server has a transaction log, located in the PGDATA/pg_xlog directory. This transaction log consists of binary snapshots of data, written to record synchronously each change to all databases' data, in case of unexpected shutdown of the database server (such as in a power failure). This ensures that data is not corrupted and no completed transaction is lost.<br />
<br />
You can also use this log to allow a copy of the original database to replicate changes made to a primary database. This was first implemented with the PITR feature in PostgreSQL 8.0, and is known as "log shipping". Log shipping is required for most forms of binary replication.<br />
<br />
This log consists of 16MB segments full of new data pages (8K segments) of the database, and not of SQL statements. For this reason there is no before and after auditing possible via this log, as you cannot know exactly what has changed. Also, the log is treated as a buffer, being deleted as it is no longer needed for crash recovery. More importantly, the data page format of the log means that log segments can only be applied to a database which is binary-identical to the database which created the log. <br />
<br />
== PITR, Warm Standby, Hot Standby, and Streaming Replication ==<br />
<br />
For the rest of this tutorial, we will refer to the active read-write instance of the Server which generates transaction logs as the "Primary" and the passive, read-only or offline instance (or instances) of the Server which receives transaction logs as the "Standby" (or "Standbys"). The term Primary/Standby is equivalent to other terminology which may be used in the database industry, such as Primary/Secondary, Primary/Replica or Master/Slave.<br />
<br />
=== PITR ===<br />
<br />
In Point-In-Time Recovery (PITR), transaction logs are copied and saved to storage until needed. Then, when needed, the Standby server can be "brought up" (made active) and transaction logs applied, either stopping when they run out or at a prior point indicated by the administrator. PITR has been available since PostgreSQL version 8.0, and as such will not be documented here.<br />
<br />
PITR is primarily used for database forensics and recovery. It is also useful when you need to back up a very large database, as it effectively supports incremental backups, which pg_dump does not.<br />
<br />
=== Warm Standby ===<br />
<br />
In Warm Standby, transaction logs are copied from the Primary and applied to the Standby immediately after they are received, or at a short delay. The Standby is offline (in "recovery mode") and not available for any query workload. This allows the Standby to be brought up to full operation very quickly. Warm Standby has been available since version 8.3, and will not be fully documented here.<br />
<br />
Warm Standby requires Log Shipping. It is primary used for database failover.<br />
<br />
=== Hot Standby ===<br />
<br />
Hot Standby is identical to Warm Standby, except that the Standby is available to run read-only queries. This offers all of the advantages of Warm Standby, plus the ability to distribute some business workload to the Standby server(s). Hot Standby by itself requires Log Shipping.<br />
<br />
Hot Standby is used both for database failover, and can also be used for load-balancing. In contrast to Streaming Replication, it places no load on the primary (except for disk space requirements) and is thus theoretically infinitely scalable. A WAL archive could be distributed to dozens or hundreds of servers via network storage. The WAL files could also easily be copied over a poor quality network connection, or by SFTP.<br />
<br />
However, since Hot Standby replicates by shipping 16MB logs, it is at best minutes behind and sometimes more than that. This can be problematic both from a failover and a load-balancing perspective.<br />
<br />
=== Streaming Replication ===<br />
<br />
Streaming Replication improves either Warm Standby or Hot Standby by opening a network connection between the Standby and the Primary database, instead of copying 16MB log files. This allows data changes to be copied over the network almost immediately on completion on the Primary. <br />
<br />
In Streaming Replication, the primary and the standby have special processes called the walsender and walreceiver which transmit modified data pages over a network port. This requires one fairly busy connection per standby, imposing an incremental load on the primary for each additional standby. Still, the load is quite low and a single primary should be able to support multiple standbys easily.<br />
<br />
Streaming replication does not require log shipping in normal operation. It may, however, require log shipping to start replication, and can utilize log shipping in order to catch up standbys which fall behind.<br />
<br />
= How to Replicate =<br />
<br />
== Cloning a Live Database ==<br />
<br />
If your workload doesn't allow you to take the primary down (and whose does?), things get a bit more complicated. You need to somehow take a "coherent snapshot" of the primary, so that you don't have an inconsistent or corrupt database on the standby. Now, in some cases this can be done via filesystem snapshotting tools or similar tricks, but as that approach is tricky and platform-dependent, we're not going to cover it here.<br />
<br />
Instead, we're going to cover the built-in method, which involves keeping a log of all changes applied to the database which happen during the copying process. The steps are essentially the same, regardless of whether you're planning to use just hot standby, streaming replication, or both. There are two parts:<br />
<br />
* Cloning the database files<br />
* Copying the archive logs<br />
<br />
Unintuitive as it is, the latter needs to be set up first, so we're going to start with that.<br />
<br />
== Setting Up Archiving On The Primary ==<br />
<br />
Archiving is the process of making an extra copy of each WAL file as it is completed. These log files then need to somehow be accessed by the standby. There are three basic ways to handle this, and you should decide in advance what method you're going to use:<br />
<br />
# Manually<br />
# Automatic file copying from primary to standby using rsync or simiar<br />
# Writing them to a common shared network file location<br />
<br />
The first method is only appropriate if you're archiving logs only to jump-start streaming replication, and you have a fairly low-traffic database or the ability to stop all writes. The third method is probably the easiest to manage if you have an appropriate network share; it can even be used to support multiple standbys with some extra thought and scripting. All of these methods will be explained below.<br />
<br />
This needs to be turned on on the primary, which if it's never been done before may require a restart (sorry, working on it), and will certainly require a reload. You'll need to set the following parameters:<br />
<br />
wal_level = hot_standby<br />
archive_mode = on<br />
archive_command = 'some command'<br />
<br />
What archive command you use depends on which archiving approach you are taking, of course. Here are three examples of commands you might use. Note that you will need to create the "archive" directories.<br />
<br />
# Manual: cp -f %p /var/lib/postgresql/data/archive/%f </dev/null<br />
# Automatic Copy: rsync -a %p 192.168.0.2:/var/lib/pgsql/data/archive/%f<br />
# Network Share: cp -f %p /shares/walarchive/archive/%f </dev/null<br />
<br />
In these commands, %p is replaced by postgres at invocation time with the full path and name of the WAL file, and %f with the name of the file alone. There are more escapes and parameters dealing with WAL archiving which will be detailed later in the tutorial. Note that, in real production, you are unlikely to want to use any commands as simple as the above. In general, you will want to have archive_command call an executable script which traps errors and can be disabled. Examples of such scripts are available in this tutorial.<br />
<br />
Now, if archive_mode was originally "off" or if you had to change wal_level, you're going to need to restart the primary (sorry, this will be fixed in a later version). If you just needed to change the archive_command, however, only a reload is required.<br />
<br />
Once you've restarted or reloaded, check the primary's logs to make sure archiving is working. If it's failing, the primary will complain extensively. You might also check that archive log files are being created; run the command "SELECT pg_switch_xlog();" as the superuser to force a new log to be written.<br />
<br />
== Setting Up Archiving on the Standby ==<br />
<br />
The standby needs to be configured to consume logs. This is simpler than the primary's setup, and doesn't really change no matter what archive copying strategy you're using.<br />
<br />
== Recovery.conf ==<br />
<br />
On the standby, replication configuration is controlled through a file called, for historical reasons, recovery.conf. If this file is present in PostgreSQL's data directory when PostgreSQL is started, that server will assume it is a standby and attempt to obey it. Generally, there is an example file installed with the other PostgreSQL shared docs. However, that example file covers all of the various replication options at once, so it's often simpler to write your own file, from scratch. Any change to recovery.conf requires a restart of the standby.<br />
<br />
In recovery.conf, you need to add a command to copy the archived WAL files to the standby's on pg_xlog directory. This is the mirror image of the archive_command on the primary. Generally, a simple cp command is sufficient:<br />
<br />
restore_command = 'cp -f /var/lib/postgresql/data/archive/%f %p </dev/null'<br />
restore_command = 'cp -f /shares/walarchive/%f %p </dev/null'<br />
<br />
Again, you might want to use a simple shell script which traps error messages, and, importantly, deletes archive files which are no longer needed. If you will be doing only hot standby and not using streaming replication, you probably want to compile the pg_standby binary provided in PostgreSQL's additional modules or "contrib", and use it instead:<br />
<br />
restore_command = 'pg_standby /shares/walarchive %f %p %r'<br />
<br />
More detail on pg_standby is in its [http://www.postgresql.org/docs/9.1/static/pgstandby.html documentation].<br />
<br />
== Cloning a Snapshot of the Primary ==<br />
<br />
Once you have archiving working, you're ready to clone the primary database. At this point, it's a simple process:<br />
<br />
# As superuser, issue the command "SELECT pg_start_backup('backup');" on the primary.<br />
# Copy all of the database files to the standby.<br />
# Start the standby database.<br />
# Issue the command "SELECT pg_stop_backup();" on the primary.<br />
<br />
Of course, each of those steps deserves a little more elaboration. pg_start_backup and pg_stop_backup are special commands you issue on the primary in order to create, hold open, and close, a "snapshot" which is how we make sure your copy of the database is not inconsistent. They also write special files to the archive log which tell the standby when it has a complete snapshot.<br />
<br />
If you are using the "manual" method of synching the archive logs, immediately after step 4 you need to do one last rsync or copy of the archive logs to the standby.<br />
<br />
When you're done with the cloning, you should see output similar to the below:<br />
<br />
This means that you're up and replicating, and should now be able to run queries on the standby.<br />
<br />
== Failing Over To The Standby ==<br />
<br />
Of course, one of the major reasons to have a standby is in case something (planned or unplanned) causes the primary server to shut down. Then you want to "fail over", or stop replication and change the standby to a full read-write primary.<br />
<br />
The recommended method is the same regardless of the type of replication or standby: via "trigger file". First, you need to set a configuration option in recovery.conf on the standby:<br />
<br />
trigger_file = '/var/lib/postgresql/data/failover'<br />
<br />
Then, when it's time to fail over, you just create an empty file with that name, such as by using the "touch" command. The standby will notice the file, attempt to apply any remaining WAL records or files it has received, and then switch to read-write or "primary" mode. When this happens, you will see a message like this in the Postgres log:<br />
<br />
PostgreSQL will also rename the recovery.conf file to recovery.done in order to prevent having the new primary fail on restart. For this reason, the recovery.conf file should be owned by the same user which the server runs as (usually "postgres").<br />
<br />
The alternative to using a trigger file is to failover manually, by deleting or renaming the recovery.conf file and restarting the standby. This method is inferior because it requires a restart which would interrupt any read-only connections to the standby currently in use.<br />
<br />
In a high-availability system, the above activity should be managed automatically in order to avoid downtime. PostgreSQL itself supplies no tools to do this, but numerous third-party utilities such as "Linux heartbeat" are compatible with PostgreSQL replication.<br />
<br />
It's important to prevent the original primary from restarting after failover, lest you end up with a "split brain" problem and data loss. There is a substantial body of literature on this, and third-party tools, so we won't discuss them here at this time.<br />
<br />
== Load Balancing ==<br />
<br />
== Managing Archive Logs ==<br />
<br />
== Tuning and Configuration of Binary Replication ==<br />
<br />
== Monitoring Replication ==<br />
<br />
<br />
[[Category:Replication]]<br />
[[Category:Howto]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Streaming_Replication&diff=37321Streaming Replication2022-11-05T01:17:57Z<p>Barwick: Update documentation link</p>
<hr />
<div>'''Streaming Replication''' (SR) provides the capability to continuously ship and<br />
apply the [http://www.postgresql.org/docs/current/static/wal.html WAL XLOG] records to some number of standby servers in order to keep them current.<br />
<br />
This feature was added to PostgreSQL 9.0. The discussion below is a developer oriented one that contains some out of date information. Users of this feature should use the documentation for the feature or a setup tutorial instead:<br />
<br />
* [https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION PostgreSQL Streaming Replication Documentation] - this documentation is for the latest version, but provides links you can use to look up the docs for your version.<br />
<br />
* [[Binary Replication Tutorial]] provides an introduction to using this replication feature.<br />
<br />
* The related but independent [[Hot Standby]] feature.<br />
<br />
<br />
= Developer and historical details on the project =<br />
SR was developed for inclusion in PostgreSQL 9.0 by NTT OSS Center. The lead developer is [mailto:masao.fujii@gmail.com Masao Fujii]. [http://www.pgcon.org/2008/schedule/events/76.en.html Synchronous Log Shipping Replication Presentation] introduces the early design of the feature.<br />
<br />
= Usage =<br />
== Users Overview ==<br />
* '''Log-shipping'''<br />
** XLOG records generated in the primary are periodically shipped to the standby via the network.<br />
** In the existing warm standby, only records in a filled file are shipped, what's referred to as file-based log-shipping. In SR, XLOG records in partially-filled XLOG file are shipped too, implementing record-based log-shipping. This means the window for data loss in SR is usually smaller than in warm standby, unless the warm standby was also configured for record-based shipping (which is complicated to setup).<br />
** The content of XLOG files written to the standby are exactly the same as those on the primary. XLOG files shipped can be used for a normal recovery and PITR.<br />
* '''Multiple standbys'''<br />
** More than one standby can establish a connection to the primary for SR. XLOG records are concurrently shipped to all these standbys. The delay/death of a standby does not harm log-shipping to other standbys.<br />
** The maximum number of standbys can be specified as a GUC variable.<br />
* '''Continuous recovery'''<br />
** The standby continuously replays XLOG records shipped without using pg_standby.<br />
** XLOG records shipped are replayed as soon as possible without waiting until XLOG file has been filled. The combination of [[Hot Standby]] and SR would make the latest data inserted into the primary visible in the standby almost immediately.<br />
** The standby periodically removes old XLOG files which are no longer needed for recovery, to prevent excessive disk usage.<br />
* '''Setup'''<br />
** The start of log-shipping does not interfere with any query processing on the primary.<br />
** The standby can be started in various conditions.<br />
*** If there are XLOG files in archive directory and restore_command is supplied, at first those files are replayed. Then the standby requests XLOG records following the last applied one to the primary. This prevents XLOG files already present in the standby from being shipped again. Similarly, XLOG files in pg_xlog are also replayed before starting log-shipping.<br />
*** If there is no XLOG files on the standby, the standby requests XLOG records following the starting XLOG location of recovery (the redo starting location).<br />
* '''Connection settings and authentication'''<br />
** A user can configure the same settings as a normal connection to a connection for SR (e.g., keepalive, pg_hba.conf).<br />
* '''Activation'''<br />
** The standby can keep waiting for activation as long as a user likes. This prevents the standby from being automatically brought up by failure of recovery or network outage.<br />
* '''Progress report'''<br />
** The primary and standby report the progress of log-shipping in PS display.<br />
* '''Graceful shutdown'''<br />
** When smart/fast shutdown is requested, the primary waits to exit until XLOG records have been sent to the standby, up to the shutdown checkpoint record.<br />
<br />
== Restrictions ==<br />
* '''Synchronous log-shipping'''<br />
** By default, SR supports operates in asynchronous manner, so the commit command might return a "success" to a client before the corresponding XLOG records are shipped to the standby. To enable synchronous replication, see [http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION Synchronous Replication]<br />
* '''Replication beyond timeline'''<br />
** A user has to get a fresh backup whenever making the old standby catch up.<br />
* '''Clustering'''<br />
** Postgres doesn't provide any clustering feature.<br />
<br />
== How to Use ==<br />
<br />
NB: there is overlap between this section and [[Binary Replication Tutorial]]<br />
<br />
* '''1.''' Install postgres in the primary and standby server as usual. This requires only ''configure'', ''make'' and ''make install''.<br />
* '''2.''' Create the initial database cluster in the primary server as usual, using ''initdb''.<br />
* '''3.''' Create an user named replication with REPLICATION privileges.<br />
$ CREATE ROLE replication WITH REPLICATION PASSWORD 'password' LOGIN;<br />
* '''4.''' Set up connections and authentication on the primary so that the standby server can successfully connect to the ''replication'' pseudo-database on the primary.<br />
<br />
$ $EDITOR postgresql.conf<br />
<br />
listen_addresses = '192.168.0.10'<br />
<br />
$ $EDITOR pg_hba.conf<br />
<br />
# The standby server must connect with a user that has replication privileges.<br />
# TYPE DATABASE USER ADDRESS METHOD<br />
host replication replication 192.168.0.20/32 md5<br />
<br />
* '''5.''' Set up the streaming replication related parameters on the primary server.<br />
$ $EDITOR postgresql.conf<br />
<br />
# To enable read-only queries on a standby server, wal_level must be set to<br />
# "hot_standby". But you can choose "archive" if you never connect to the<br />
# server in standby mode.<br />
wal_level = hot_standby<br />
<br />
# Set the maximum number of concurrent connections from the standby servers.<br />
max_wal_senders = 5<br />
<br />
# To prevent the primary server from removing the WAL segments required for<br />
# the standby server before shipping them, set the minimum number of segments<br />
# retained in the pg_xlog directory. At least wal_keep_segments should be<br />
# larger than the number of segments generated between the beginning of<br />
# online-backup and the startup of streaming replication. If you enable WAL<br />
# archiving to an archive directory accessible from the standby, this may<br />
# not be necessary.<br />
wal_keep_segments = 32<br />
<br />
# Enable WAL archiving on the primary to an archive directory accessible from<br />
# the standby. If wal_keep_segments is a high enough number to retain the WAL<br />
# segments required for the standby server, this is not necessary.<br />
archive_mode = on<br />
archive_command = 'cp %p /path_to/archive/%f'<br />
* '''6.''' Start postgres on the primary server.<br />
* '''7.''' Make a base backup by copying the primary server's data directory to the standby server.<br />
<br />
** '''7.1.''' Do it with pg_(start|stop)_backup and rsync on the primary<br />
<br />
$ psql -c "SELECT pg_start_backup('label', true)"<br />
$ rsync -ac ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid<br />
$ psql -c "SELECT pg_stop_backup()"<br />
<br />
** '''7.2.''' Do it with pg_basebackup on the standby<br />
<br />
In version 9.1+, pg_basebackup can do the dirty work of fetching the entire data directory of your PostgreSQL installation from the primary and placing it onto the standby server.<br />
<br />
The prerequisite is that you make sure the standby's data directory is empty.<br />
<br />
Make sure to remove any tablespace directories as well. You can find those directories with:<br />
<br />
$ psql -c '\db'<br />
<br />
If you keep your postgresql.conf and other config files in PGDATA, you need a backup of postgresql.conf, to restore after pg_basebackup.<br />
<br />
After you've cleared all the directories, you can use the following command to directly stream the data from the primary onto your standby server.<br />
Run it as the database superuser, typically 'postgres', to make sure the permissions are preserved (use su, sudo or whatever other tool to make sure you're not root).<br />
<br />
$ pg_basebackup -h 192.168.0.10 -D /srv/pgsql/standby -P -U replication --xlog-method=stream<br />
<br />
In version 9.3+, you can also add the -R option so it creates a minimal recovery command file for step 9 below.<br />
<br />
If you backed up postgresql.conf, now restore it.<br />
<br />
* '''8.''' Set up replication-related parameters, connections and authentication in the standby server like the primary, so that the standby might work as a primary after failover.<br />
* '''9.''' Enable read-only queries on the standby server. But if wal_level is ''archive'' on the primary, leave hot_standby unchanged (i.e., off).<br />
$ $EDITOR postgresql.conf<br />
<br />
hot_standby = on<br />
* '''10.''' Create a recovery command file in the standby server; the following parameters are required for streaming replication.<br />
$ $EDITOR recovery.conf<br />
# Note that recovery.conf must be in the $PGDATA directory, even if the<br />
# main postgresql.conf file is located elsewhere.<br />
<br />
# Specifies whether to start the server as a standby. In streaming replication,<br />
# this parameter must to be set to on.<br />
standby_mode = 'on'<br />
<br />
# Specifies a connection string which is used for the standby server to connect<br />
# with the primary.<br />
primary_conninfo = 'host=192.168.0.10 port=5432 user=replication password=password'<br />
<br />
# Specifies a trigger file whose presence should cause streaming replication to<br />
# end (i.e., failover).<br />
trigger_file = '/path_to/trigger'<br />
<br />
# Specifies a command to load archive segments from the WAL archive. If<br />
# wal_keep_segments is a high enough number to retain the WAL segments<br />
# required for the standby server, this may not be necessary. But<br />
# a large workload can cause segments to be recycled before the standby<br />
# is fully synchronized, requiring you to start again from a new base backup.<br />
restore_command = 'cp /path_to/archive/%f "%p"'<br />
* '''11.''' Start postgres in the standby server. It will start streaming replication.<br />
* '''12.''' You can calculate the replication lag by comparing the current WAL write location on the primary with the last WAL location received/replayed by the standby. They can be retrieved using ''pg_current_xlog_location'' on the primary and the ''pg_last_xlog_receive_location''/''pg_last_xlog_replay_location'' on the standby, respectively.<br />
$ psql -c "SELECT pg_current_xlog_location()" -h192.168.0.10 (primary host)<br />
pg_current_xlog_location <br />
--------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_receive_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_receive_location <br />
-------------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_replay_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_replay_location <br />
------------------------------<br />
0/2000000<br />
(1 row)<br />
* '''13.''' You can also check the progress of streaming replication by using ''ps'' command.<br />
# The displayed LSNs indicate the byte position that the standby server has<br />
# written up to in the xlogs.<br />
[primary] $ ps -ef | grep sender<br />
postgres 6879 6831 0 10:31 ? 00:00:00 postgres: wal sender process postgres 127.0.0.1(44663) streaming 0/2000000<br />
<br />
[standby] $ ps -ef | grep receiver<br />
postgres 6878 6872 1 10:31 ? 00:00:01 postgres: wal receiver process streaming 0/2000000<br />
* How to do failover<br />
** Create the trigger file in the standby after the primary fails.<br />
* How to stop the primary or the standby server<br />
** Shut down it as usual (''pg_ctl stop'').<br />
* How to restart streaming replication after failover<br />
** Repeat the operations from '''6th'''; making a fresh backup, some configurations and starting the original primary as the standby. The primary server doesn't need to be stopped during these operations.<br />
* How to restart streaming replication after the standby fails<br />
** Restart postgres in the standby server after eliminating the cause of failure.<br />
* How to disconnect the standby from the primary<br />
** Create the trigger file in the standby while the primary is running. Then the standby would be brought up.<br />
* How to re-synchronize the stand-alone standby after isolation<br />
** Shut down the standby as usual. And repeat the operations from '''6th'''.<br />
* If you have more than one standby, promoting one will break the other(s). Update their recovery.conf settings to point to the new master, set recovery_target_timeline to 'latest', scp/rsync the pg_xlog directory, and restart the standby.<br />
<br />
= Todo =<br />
== v9.0 ==<br />
<br />
Moved to [[PostgreSQL_9.0_Open_Items]]<br />
<br />
=== Committed ===<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01455.php Retrying from archive and some refactoring around Read/FetchRecord().] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00395.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg02601.php SR wrongly treats the WAL-boundary.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00396.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01715.php Adjust SR for some later changes about wal-skipping.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00399.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00024.php VACUUM FULL unexpectedly writes an XLOG UNLOGGED record.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00038.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01754.php Add a message type header.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00037.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01536.php Documentation: Add a new "Replication" chapter.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00115.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00350.php Failed assertion during recovery of partial WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00124.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00712.php A PANIC error might occur in the standby because of a partially-filled archived WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00137.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00330.php Improve the standby messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00140.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01672.php pq_getbyte_if_available() is not working because the win32 socket emulation layer simply wasn't designed to deal with non-blocking sockets.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00198.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01488.php Walsender might emit unfit messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00239.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01236.php Streaming replication on win32, still broken.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00270.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00992.php Create new section for recovery.conf.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00295.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01824.php Assertion failure in walreceiver.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00356.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01717.php Forbid a startup of walsender during recovery, and emit a suitable message? Or allow walsender to be started also during recovery?] - [http://archives.postgresql.org/message-id/20100316090955.9A5107541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01003.php How do we clean down the archive without using pg_standby?] - [http://archives.postgresql.org/message-id/20100318091718.BC14D7541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01510.php File-based log shipping without pg_standby doesn't replay the WAL files in pg_xlog.] - [http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php commit]<br />
<br />
== v9.1 ==<br />
=== Synchronization capability ===<br />
* Introduce the replication mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'' and ''fsync''.<br />
** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
** ''recv'' or ''fsync'' makes transaction commit wait for XLOG to be received or fsynced by the standby, respectively.<br />
** (''apply'' makes transaction commit wait for XLOG to be replayed by the standby. This mode will be supported in v9.2 or later)<br />
** The replication mode is specified in recovery.conf of the standby as well as other parameters for replication.<br />
*** The startup process reads the replication mode from recovery.conf and shares it to walreceiver via new shared-memory variable.<br />
*** Walreceiver also shares it to walsender by using the replication handshake message (existing protocol needs to be extended).<br />
** Based on the replication mode, walreceiver sends the reply meaning that replication is done up to the specified location to the primary.<br />
*** In async, walreceiver doesn't need to send any reply other than end-of-replication message.<br />
*** In recv or fsync, walreceiver sends the reply just after receiving or flushing XLOG, respectively.<br />
*** New message type for the reply needs to be defined. The reply is sent as CopyData message.<br />
** Walreceiver writes all the outstanding XLOG to disk before shutting down.<br />
** Walsender receives the reply from the standby, updates the location of the last record replicated, and announces completion of replication.<br />
*** New shared-memory variable to keep that location is required.<br />
** When processing the commit command, backend waits for XLOG to be replicated to only the standbys which are in the recv or fsync replication mode.<br />
*** Also smart shutdown waits for XLOG of shutdown checkpoint to be replicated.<br />
* Required optimization<br />
** Walsender should send outstanding XLOG without waiting wal_sender_delay.<br />
*** When processing the commit command, backend signals walsender to send outstanding XLOG immediately.<br />
** Backend should exit the wait loop as soon as the reply arrives at the primary.<br />
*** When receiving the reply, walsender signals backends to get up from the sleep and determine whether to exit the wait loop by checking the location of the last XLOG replicated.<br />
*** Only backends waiting for XLOG to be replicated up to the location contained in the reply are sent the signal.<br />
** Walsender waits for the signal from backends and the reply from the standby at the same time, by using select/poll.<br />
** Walsender reads XLOG from not only disk but also shared memory (wal buffers).<br />
** Walreceiver should flush XLOG file only when XLOG file is switched or the related page is flushed.<br />
*** When startup process or bgwriter flushes the buffer page, it checks whether the related XLOG has already been flushed via shared memory (location of the last XLOG flushed).<br />
*** It flushes the buffer page, if XLOG file has already been flushed.<br />
*** It signals walreceiver to flush XLOG file immediately and waits for the flush to complete, if XLOG file has not been flushed yet.<br />
** While the standby is catching up with the primary, those servers should ignore the replication mode and perform asynchronous replication.<br />
*** After those servers have almost gotten into synchronization, they perform replication based on the specified replication mode.<br />
*** New replication states like 'catching-up', 'sync', etc need to be defined, and the state machine for them is required on both servers.<br />
*** Current replication state can be monitored on both servers via SQL.<br />
* Required timeout<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG is replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
<br />
== Future release ==<br />
* '''Synchronization capability'''<br />
** Introduce the synchronization mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'', ''fsync'' and ''apply''.<br />
*** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
*** ''recv'', ''fsync'' and ''apply'' makes transaction commit wait for XLOG records to be received, fsynced and applied on the standby, respectively.<br />
** Change walsender to be able to read XLOG from not only the disk but also shared memory.<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG records are replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
* '''Monitoring'''<br />
** Provide the capability to check the progress and gap of streaming replication via one query. A collaboration of HS and SR is necessary to provide that capability on the standby side.<br />
** Provide the capability to check if the specified repliation is in progress via a query. Also more detailed status information might be necessary, e.g, the standby is catching up now, has already gotten into sync, and so on.<br />
** Change the stats collector to collect the statistics information about replication, e.g., average delay of replication time.<br />
** Develop the tool to calculate the latest XLOG position from XLOG files. This is necessary to check the gap of replication after the server fails.<br />
** Also develop the tool to extract the user-readable contents from XLOG files. This is necessary to see the contents of the gap, and manually restore them.<br />
* '''Easy to Use'''<br />
** Introduce the parameters like:<br />
*** replication_halt_timeout - replication will halt if no data has been sent for this much time.<br />
*** replication_halt_segments - replication will halt if number of WAL files in pg_xlog exceeds this threshold.<br />
*** These parameters allow us to avoid disk overflow.<br />
** Add new feature which transfers also base backup via the direct connection between the primary and the standby.<br />
** Add new hooks like walsender_hook and walreceiver_hook to cooperate with the add-on program for compression like pglesslog.<br />
** Provide a graceful termination of replication via a query on the primary. On the standby, a trigger file mechanism already provides that capability.<br />
** Support replication beyond timeline. The timeline history files need to be shipped from the primary to the standby.<br />
* '''Robustness'''<br />
** Support keepalive in libpq. This is useful for a client and the standby to detect a failure of the primary immediately.<br />
* '''Miscellaneous'''<br />
** Standalone walreceiver tool, which connects to the primary, continuously receives and writes XLOG records, independently from postgres server.<br />
** Cascade streaming replication. Allow walsender to send XLOG to another standby during recovery.<br />
** WAL archiving during recovery.<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Streaming_Replication&diff=37320Streaming Replication2022-11-05T01:11:54Z<p>Barwick: Clarify location of recovery.conf</p>
<hr />
<div>'''Streaming Replication''' (SR) provides the capability to continuously ship and<br />
apply the [http://www.postgresql.org/docs/current/static/wal.html WAL XLOG] records to some number of standby servers in order to keep them current.<br />
<br />
This feature was added to PostgreSQL 9.0. The discussion below is a developer oriented one that contains some out of date information. Users of this feature should use the documentation for the feature or a setup tutorial instead:<br />
<br />
* [http://www.postgresql.org/docs/current/static/warm-standby.html PostgreSQL Streaming Replication Documentation] - this documentation is for the latest version, but provides links you can use to look up the docs for your version.<br />
<br />
* [[Binary Replication Tutorial]] provides an introduction to using this replication feature.<br />
<br />
* The related but independent [[Hot Standby]] feature.<br />
<br />
<br />
= Developer and historical details on the project =<br />
SR was developed for inclusion in PostgreSQL 9.0 by NTT OSS Center. The lead developer is [mailto:masao.fujii@gmail.com Masao Fujii]. [http://www.pgcon.org/2008/schedule/events/76.en.html Synchronous Log Shipping Replication Presentation] introduces the early design of the feature.<br />
<br />
= Usage =<br />
== Users Overview ==<br />
* '''Log-shipping'''<br />
** XLOG records generated in the primary are periodically shipped to the standby via the network.<br />
** In the existing warm standby, only records in a filled file are shipped, what's referred to as file-based log-shipping. In SR, XLOG records in partially-filled XLOG file are shipped too, implementing record-based log-shipping. This means the window for data loss in SR is usually smaller than in warm standby, unless the warm standby was also configured for record-based shipping (which is complicated to setup).<br />
** The content of XLOG files written to the standby are exactly the same as those on the primary. XLOG files shipped can be used for a normal recovery and PITR.<br />
* '''Multiple standbys'''<br />
** More than one standby can establish a connection to the primary for SR. XLOG records are concurrently shipped to all these standbys. The delay/death of a standby does not harm log-shipping to other standbys.<br />
** The maximum number of standbys can be specified as a GUC variable.<br />
* '''Continuous recovery'''<br />
** The standby continuously replays XLOG records shipped without using pg_standby.<br />
** XLOG records shipped are replayed as soon as possible without waiting until XLOG file has been filled. The combination of [[Hot Standby]] and SR would make the latest data inserted into the primary visible in the standby almost immediately.<br />
** The standby periodically removes old XLOG files which are no longer needed for recovery, to prevent excessive disk usage.<br />
* '''Setup'''<br />
** The start of log-shipping does not interfere with any query processing on the primary.<br />
** The standby can be started in various conditions.<br />
*** If there are XLOG files in archive directory and restore_command is supplied, at first those files are replayed. Then the standby requests XLOG records following the last applied one to the primary. This prevents XLOG files already present in the standby from being shipped again. Similarly, XLOG files in pg_xlog are also replayed before starting log-shipping.<br />
*** If there is no XLOG files on the standby, the standby requests XLOG records following the starting XLOG location of recovery (the redo starting location).<br />
* '''Connection settings and authentication'''<br />
** A user can configure the same settings as a normal connection to a connection for SR (e.g., keepalive, pg_hba.conf).<br />
* '''Activation'''<br />
** The standby can keep waiting for activation as long as a user likes. This prevents the standby from being automatically brought up by failure of recovery or network outage.<br />
* '''Progress report'''<br />
** The primary and standby report the progress of log-shipping in PS display.<br />
* '''Graceful shutdown'''<br />
** When smart/fast shutdown is requested, the primary waits to exit until XLOG records have been sent to the standby, up to the shutdown checkpoint record.<br />
<br />
== Restrictions ==<br />
* '''Synchronous log-shipping'''<br />
** By default, SR supports operates in asynchronous manner, so the commit command might return a "success" to a client before the corresponding XLOG records are shipped to the standby. To enable synchronous replication, see [http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION Synchronous Replication]<br />
* '''Replication beyond timeline'''<br />
** A user has to get a fresh backup whenever making the old standby catch up.<br />
* '''Clustering'''<br />
** Postgres doesn't provide any clustering feature.<br />
<br />
== How to Use ==<br />
<br />
NB: there is overlap between this section and [[Binary Replication Tutorial]]<br />
<br />
* '''1.''' Install postgres in the primary and standby server as usual. This requires only ''configure'', ''make'' and ''make install''.<br />
* '''2.''' Create the initial database cluster in the primary server as usual, using ''initdb''.<br />
* '''3.''' Create an user named replication with REPLICATION privileges.<br />
$ CREATE ROLE replication WITH REPLICATION PASSWORD 'password' LOGIN;<br />
* '''4.''' Set up connections and authentication on the primary so that the standby server can successfully connect to the ''replication'' pseudo-database on the primary.<br />
<br />
$ $EDITOR postgresql.conf<br />
<br />
listen_addresses = '192.168.0.10'<br />
<br />
$ $EDITOR pg_hba.conf<br />
<br />
# The standby server must connect with a user that has replication privileges.<br />
# TYPE DATABASE USER ADDRESS METHOD<br />
host replication replication 192.168.0.20/32 md5<br />
<br />
* '''5.''' Set up the streaming replication related parameters on the primary server.<br />
$ $EDITOR postgresql.conf<br />
<br />
# To enable read-only queries on a standby server, wal_level must be set to<br />
# "hot_standby". But you can choose "archive" if you never connect to the<br />
# server in standby mode.<br />
wal_level = hot_standby<br />
<br />
# Set the maximum number of concurrent connections from the standby servers.<br />
max_wal_senders = 5<br />
<br />
# To prevent the primary server from removing the WAL segments required for<br />
# the standby server before shipping them, set the minimum number of segments<br />
# retained in the pg_xlog directory. At least wal_keep_segments should be<br />
# larger than the number of segments generated between the beginning of<br />
# online-backup and the startup of streaming replication. If you enable WAL<br />
# archiving to an archive directory accessible from the standby, this may<br />
# not be necessary.<br />
wal_keep_segments = 32<br />
<br />
# Enable WAL archiving on the primary to an archive directory accessible from<br />
# the standby. If wal_keep_segments is a high enough number to retain the WAL<br />
# segments required for the standby server, this is not necessary.<br />
archive_mode = on<br />
archive_command = 'cp %p /path_to/archive/%f'<br />
* '''6.''' Start postgres on the primary server.<br />
* '''7.''' Make a base backup by copying the primary server's data directory to the standby server.<br />
<br />
** '''7.1.''' Do it with pg_(start|stop)_backup and rsync on the primary<br />
<br />
$ psql -c "SELECT pg_start_backup('label', true)"<br />
$ rsync -ac ${PGDATA}/ standby:/srv/pgsql/standby/ --exclude postmaster.pid<br />
$ psql -c "SELECT pg_stop_backup()"<br />
<br />
** '''7.2.''' Do it with pg_basebackup on the standby<br />
<br />
In version 9.1+, pg_basebackup can do the dirty work of fetching the entire data directory of your PostgreSQL installation from the primary and placing it onto the standby server.<br />
<br />
The prerequisite is that you make sure the standby's data directory is empty.<br />
<br />
Make sure to remove any tablespace directories as well. You can find those directories with:<br />
<br />
$ psql -c '\db'<br />
<br />
If you keep your postgresql.conf and other config files in PGDATA, you need a backup of postgresql.conf, to restore after pg_basebackup.<br />
<br />
After you've cleared all the directories, you can use the following command to directly stream the data from the primary onto your standby server.<br />
Run it as the database superuser, typically 'postgres', to make sure the permissions are preserved (use su, sudo or whatever other tool to make sure you're not root).<br />
<br />
$ pg_basebackup -h 192.168.0.10 -D /srv/pgsql/standby -P -U replication --xlog-method=stream<br />
<br />
In version 9.3+, you can also add the -R option so it creates a minimal recovery command file for step 9 below.<br />
<br />
If you backed up postgresql.conf, now restore it.<br />
<br />
* '''8.''' Set up replication-related parameters, connections and authentication in the standby server like the primary, so that the standby might work as a primary after failover.<br />
* '''9.''' Enable read-only queries on the standby server. But if wal_level is ''archive'' on the primary, leave hot_standby unchanged (i.e., off).<br />
$ $EDITOR postgresql.conf<br />
<br />
hot_standby = on<br />
* '''10.''' Create a recovery command file in the standby server; the following parameters are required for streaming replication.<br />
$ $EDITOR recovery.conf<br />
# Note that recovery.conf must be in the $PGDATA directory, even if the<br />
# main postgresql.conf file is located elsewhere.<br />
<br />
# Specifies whether to start the server as a standby. In streaming replication,<br />
# this parameter must to be set to on.<br />
standby_mode = 'on'<br />
<br />
# Specifies a connection string which is used for the standby server to connect<br />
# with the primary.<br />
primary_conninfo = 'host=192.168.0.10 port=5432 user=replication password=password'<br />
<br />
# Specifies a trigger file whose presence should cause streaming replication to<br />
# end (i.e., failover).<br />
trigger_file = '/path_to/trigger'<br />
<br />
# Specifies a command to load archive segments from the WAL archive. If<br />
# wal_keep_segments is a high enough number to retain the WAL segments<br />
# required for the standby server, this may not be necessary. But<br />
# a large workload can cause segments to be recycled before the standby<br />
# is fully synchronized, requiring you to start again from a new base backup.<br />
restore_command = 'cp /path_to/archive/%f "%p"'<br />
* '''11.''' Start postgres in the standby server. It will start streaming replication.<br />
* '''12.''' You can calculate the replication lag by comparing the current WAL write location on the primary with the last WAL location received/replayed by the standby. They can be retrieved using ''pg_current_xlog_location'' on the primary and the ''pg_last_xlog_receive_location''/''pg_last_xlog_replay_location'' on the standby, respectively.<br />
$ psql -c "SELECT pg_current_xlog_location()" -h192.168.0.10 (primary host)<br />
pg_current_xlog_location <br />
--------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_receive_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_receive_location <br />
-------------------------------<br />
0/2000000<br />
(1 row)<br />
<br />
$ psql -c "select pg_last_xlog_replay_location()" -h192.168.0.20 (standby host)<br />
pg_last_xlog_replay_location <br />
------------------------------<br />
0/2000000<br />
(1 row)<br />
* '''13.''' You can also check the progress of streaming replication by using ''ps'' command.<br />
# The displayed LSNs indicate the byte position that the standby server has<br />
# written up to in the xlogs.<br />
[primary] $ ps -ef | grep sender<br />
postgres 6879 6831 0 10:31 ? 00:00:00 postgres: wal sender process postgres 127.0.0.1(44663) streaming 0/2000000<br />
<br />
[standby] $ ps -ef | grep receiver<br />
postgres 6878 6872 1 10:31 ? 00:00:01 postgres: wal receiver process streaming 0/2000000<br />
* How to do failover<br />
** Create the trigger file in the standby after the primary fails.<br />
* How to stop the primary or the standby server<br />
** Shut down it as usual (''pg_ctl stop'').<br />
* How to restart streaming replication after failover<br />
** Repeat the operations from '''6th'''; making a fresh backup, some configurations and starting the original primary as the standby. The primary server doesn't need to be stopped during these operations.<br />
* How to restart streaming replication after the standby fails<br />
** Restart postgres in the standby server after eliminating the cause of failure.<br />
* How to disconnect the standby from the primary<br />
** Create the trigger file in the standby while the primary is running. Then the standby would be brought up.<br />
* How to re-synchronize the stand-alone standby after isolation<br />
** Shut down the standby as usual. And repeat the operations from '''6th'''.<br />
* If you have more than one standby, promoting one will break the other(s). Update their recovery.conf settings to point to the new master, set recovery_target_timeline to 'latest', scp/rsync the pg_xlog directory, and restart the standby.<br />
<br />
= Todo =<br />
== v9.0 ==<br />
<br />
Moved to [[PostgreSQL_9.0_Open_Items]]<br />
<br />
=== Committed ===<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01455.php Retrying from archive and some refactoring around Read/FetchRecord().] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00395.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg02601.php SR wrongly treats the WAL-boundary.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00396.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01715.php Adjust SR for some later changes about wal-skipping.] - [http://archives.postgresql.org/pgsql-committers/2010-01/msg00399.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00024.php VACUUM FULL unexpectedly writes an XLOG UNLOGGED record.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00038.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01754.php Add a message type header.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00037.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01536.php Documentation: Add a new "Replication" chapter.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00115.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00350.php Failed assertion during recovery of partial WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00124.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00712.php A PANIC error might occur in the standby because of a partially-filled archived WAL file.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00137.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00330.php Improve the standby messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00140.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01672.php pq_getbyte_if_available() is not working because the win32 socket emulation layer simply wasn't designed to deal with non-blocking sockets.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00198.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01488.php Walsender might emit unfit messages.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00239.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01236.php Streaming replication on win32, still broken.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00270.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg00992.php Create new section for recovery.conf.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00295.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01824.php Assertion failure in walreceiver.] - [http://archives.postgresql.org/pgsql-committers/2010-02/msg00356.php commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-01/msg01717.php Forbid a startup of walsender during recovery, and emit a suitable message? Or allow walsender to be started also during recovery?] - [http://archives.postgresql.org/message-id/20100316090955.9A5107541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01003.php How do we clean down the archive without using pg_standby?] - [http://archives.postgresql.org/message-id/20100318091718.BC14D7541D0@cvs.postgresql.org commit]<br />
* [http://archives.postgresql.org/pgsql-hackers/2010-02/msg01510.php File-based log shipping without pg_standby doesn't replay the WAL files in pg_xlog.] - [http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php commit]<br />
<br />
== v9.1 ==<br />
=== Synchronization capability ===<br />
* Introduce the replication mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'' and ''fsync''.<br />
** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
** ''recv'' or ''fsync'' makes transaction commit wait for XLOG to be received or fsynced by the standby, respectively.<br />
** (''apply'' makes transaction commit wait for XLOG to be replayed by the standby. This mode will be supported in v9.2 or later)<br />
** The replication mode is specified in recovery.conf of the standby as well as other parameters for replication.<br />
*** The startup process reads the replication mode from recovery.conf and shares it to walreceiver via new shared-memory variable.<br />
*** Walreceiver also shares it to walsender by using the replication handshake message (existing protocol needs to be extended).<br />
** Based on the replication mode, walreceiver sends the reply meaning that replication is done up to the specified location to the primary.<br />
*** In async, walreceiver doesn't need to send any reply other than end-of-replication message.<br />
*** In recv or fsync, walreceiver sends the reply just after receiving or flushing XLOG, respectively.<br />
*** New message type for the reply needs to be defined. The reply is sent as CopyData message.<br />
** Walreceiver writes all the outstanding XLOG to disk before shutting down.<br />
** Walsender receives the reply from the standby, updates the location of the last record replicated, and announces completion of replication.<br />
*** New shared-memory variable to keep that location is required.<br />
** When processing the commit command, backend waits for XLOG to be replicated to only the standbys which are in the recv or fsync replication mode.<br />
*** Also smart shutdown waits for XLOG of shutdown checkpoint to be replicated.<br />
* Required optimization<br />
** Walsender should send outstanding XLOG without waiting wal_sender_delay.<br />
*** When processing the commit command, backend signals walsender to send outstanding XLOG immediately.<br />
** Backend should exit the wait loop as soon as the reply arrives at the primary.<br />
*** When receiving the reply, walsender signals backends to get up from the sleep and determine whether to exit the wait loop by checking the location of the last XLOG replicated.<br />
*** Only backends waiting for XLOG to be replicated up to the location contained in the reply are sent the signal.<br />
** Walsender waits for the signal from backends and the reply from the standby at the same time, by using select/poll.<br />
** Walsender reads XLOG from not only disk but also shared memory (wal buffers).<br />
** Walreceiver should flush XLOG file only when XLOG file is switched or the related page is flushed.<br />
*** When startup process or bgwriter flushes the buffer page, it checks whether the related XLOG has already been flushed via shared memory (location of the last XLOG flushed).<br />
*** It flushes the buffer page, if XLOG file has already been flushed.<br />
*** It signals walreceiver to flush XLOG file immediately and waits for the flush to complete, if XLOG file has not been flushed yet.<br />
** While the standby is catching up with the primary, those servers should ignore the replication mode and perform asynchronous replication.<br />
*** After those servers have almost gotten into synchronization, they perform replication based on the specified replication mode.<br />
*** New replication states like 'catching-up', 'sync', etc need to be defined, and the state machine for them is required on both servers.<br />
*** Current replication state can be monitored on both servers via SQL.<br />
* Required timeout<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG is replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
<br />
== Future release ==<br />
* '''Synchronization capability'''<br />
** Introduce the synchronization mode which can control how long transaction commit waits for replication before the commit command returns a "success" to a client. The valid modes are ''async'', ''recv'', ''fsync'' and ''apply''.<br />
*** ''async'' doesn't make transaction commit wait for replication, i.e., asynchronous replication.<br />
*** ''recv'', ''fsync'' and ''apply'' makes transaction commit wait for XLOG records to be received, fsynced and applied on the standby, respectively.<br />
** Change walsender to be able to read XLOG from not only the disk but also shared memory.<br />
** Add new parameter replication_timeout which is the maximum time to wait until XLOG records are replicated to the standby. (does this match http://www.postgresql.org/docs/current/interactive/runtime-config-replication.html ?)<br />
** Add new parameter (replication_timeout_action) to specify the reaction to replication_timeout.<br />
* '''Monitoring'''<br />
** Provide the capability to check the progress and gap of streaming replication via one query. A collaboration of HS and SR is necessary to provide that capability on the standby side.<br />
** Provide the capability to check if the specified repliation is in progress via a query. Also more detailed status information might be necessary, e.g, the standby is catching up now, has already gotten into sync, and so on.<br />
** Change the stats collector to collect the statistics information about replication, e.g., average delay of replication time.<br />
** Develop the tool to calculate the latest XLOG position from XLOG files. This is necessary to check the gap of replication after the server fails.<br />
** Also develop the tool to extract the user-readable contents from XLOG files. This is necessary to see the contents of the gap, and manually restore them.<br />
* '''Easy to Use'''<br />
** Introduce the parameters like:<br />
*** replication_halt_timeout - replication will halt if no data has been sent for this much time.<br />
*** replication_halt_segments - replication will halt if number of WAL files in pg_xlog exceeds this threshold.<br />
*** These parameters allow us to avoid disk overflow.<br />
** Add new feature which transfers also base backup via the direct connection between the primary and the standby.<br />
** Add new hooks like walsender_hook and walreceiver_hook to cooperate with the add-on program for compression like pglesslog.<br />
** Provide a graceful termination of replication via a query on the primary. On the standby, a trigger file mechanism already provides that capability.<br />
** Support replication beyond timeline. The timeline history files need to be shipped from the primary to the standby.<br />
* '''Robustness'''<br />
** Support keepalive in libpq. This is useful for a client and the standby to detect a failure of the primary immediately.<br />
* '''Miscellaneous'''<br />
** Standalone walreceiver tool, which connects to the primary, continuously receives and writes XLOG records, independently from postgres server.<br />
** Cascade streaming replication. Allow walsender to send XLOG to another standby during recovery.<br />
** WAL archiving during recovery.<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=37287Repmgr2022-10-18T01:25:46Z<p>Barwick: Add new release</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL replication and failover management, introduced by [https://2ndQuadrant.com 2ndQuadrant] (now EDB) in 2010. <br />
<br />
repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements. It helps DBAs and System Administrators manage a cluster of PostgreSQL databases by taking advantage of the Hot Standby capability introduced in PostgreSQL 9.<br />
<br />
repmgr streamlines the process of administration and daily management, enhances productivity, complements built-in replication capabilities, and reduces overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process.<br />
* providing support for HA operations such as switchovers and failovers.<br />
<br />
repmgr is available via the EDB package repositories as well as the PGDG community repositories. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 5 is compatible with PostgreSQL 9.4 and above. See the [https://repmgr.org/docs/current/install-requirements.html#INSTALL-COMPATIBILITY-MATRIX repmgr compatibility matrix] for full details.<br />
<br />
<br />
== repmgr 5 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/current/release-5.3.3.html 5.3.3] (2022-10-17).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or Barman<br />
* Timeline following:<br />
** a standby that can be promoted to a primary without requiring a restart<br />
** other standbys that can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the primary to another standby mode<br />
* Replication slot support , simplifying WAL retention management<br />
* [https://repmgr.org/docs/current/performing-switchover.html Switchover] support for role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at https://github.com/EnterpriseDB/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Package installation instructions for Red Hat/Debian-based distributions can be found [https://repmgr.org/docs/current/installation-packages.html here].<br />
<br />
== Documentation ==<br />
<br />
* [https://repmgr.org/docs/current/index.html repmgr documentation]<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=36760Foreign data wrappers2022-02-20T05:21:05Z<p>Barwick: /* Specific SQL Database Wrappers */ new firebird_fdw version</p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained ?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| [https://gdal.org/drivers/vector/index.html GDAL/OGR]<br />
| Native<br />
| MIT<br />
| [https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
| yum.postgresql.org, apt.postgresql.org, and part of PostGIS windows bundle (application stackbuilder)<br />
| <br />
| Can access many kinds of data sources (Relational databases, spreadsheets, CSV files, web feature services, etc). Uses the [https://gdal.org/ GDAL library] which supports hundreds of formats to access the data. Exposes vector data as PostGIS geometry columns if you have PostGIS installed. Works great with both spatial and non-spatial data.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|<br />
|-<br />
| APIs (via [https://hub.steampipe.io/plugins Steampipe plugins])<br />
| Native<br />
| CLI and FDW extension: AGPL, Plugins: Apache 2.0<br />
| [https://github.com/turbot/steampipe CLI on GitHub], [https://github.com/turbot/steampipe-postgres-fdw FDW extension on GitHub]<br />
| [https://steampipe.io/downloads Steampipe downloads]<br />
| [https://steampipe.io/docs Steampipe docs]<br />
| [https://steampipe.io Steampipe] bundles Postgres with an FDW extension that supports a growing ecosystem of plugins. The plugins consume APIs, map them to tables, and enable queries within and across APIs.<br />
|}<br />
<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/postgres-fdw.html documentation]<br />
|<br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
|<br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|FDW for MySQL<br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|DB2<br />
|Native<br />
|<br />
|[https://github.com/wolfgangbrandl/db2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.firebirdsql.org/ Firebird]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|[https://github.com/ibarwick/firebird_fdw/blob/master/README.md README]<br />
|version [https://github.com/ibarwick/firebird_fdw/releases/tag/1.2.3 1.2.3] released (2022-02)<br />
|-<br />
|[https://www.sqlite.org/index.html SQLite]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pgspider/sqlite_fdw github]<br />
|[https://pgxn.org/dist/sqlite_fdw PGXN]<br />
|[https://github.com/pgspider/sqlite_fdw/blob/master/README.md README]<br />
|An FDW for SQLite3 (write support and several pushdown optimization)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[https://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[https://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|Native<br />
|Apache<br />
|[https://github.com/adjust/clickhouse_fdw Github]<br />
|<br />
|[https://github.com/adjust/clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|Native<br />
|<br />
|[https://github.com/Percona-Lab/clickhousedb_fdw Github]<br />
|<br />
|[https://github.com/Percona-Lab/clickhousedb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
| [https://github.com/griddb/griddb_nosql GridDB]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/griddb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/griddb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| InfluxDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/influxdb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/influxdb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| [https://kafka.apache.org/ Kafka]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/kafka_fdw GitHub]<br />
|<br />
| [https://github.com/adjust/kafka_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|[https://multicorn.org/ Multicorn]<br />
|GPLv3<br />
|[https://github.com/sim51/neo4j-fdw Github]<br />
|<br />
|[https://github.com/sim51/neo4j-fdw/blob/master/README.adoc README]<br />
|FWD for Neo4j and also add a Cypher function to Pg<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/slamdata/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://github.com/facebook/rocksdb RocksDB]<br />
|Native<br />
|Apache<br />
|[https://github.com/vidardb/PostgresForeignDataWrapper Github]<br />
|<br />
|[https://github.com/vidardb/PostgresForeignDataWrapper/blob/master/README.md README]<br />
|FDW for RocksDB<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to]<br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| CSV / gzipped<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
|<br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
|<br />
|-<br />
| Multi-File<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| Parquet<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/parquet_fdw GitHub]<br />
|<br />
|<br />
| Foreign data wrapper for reading Parquet files using libarrow/libparquet<br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| XML<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [https://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [https://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [https://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
|<br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
|<br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
|<br />
|}<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ICAL<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
|<br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
|<br />
|-<br />
| IMAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
|<br />
|-<br />
| RSS<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed.<br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|-<br />
| pgsql-http<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pramsey/pgsql-http GitHub]<br />
| Compile<br />
| <br />
| Allows to query any http resource using CURL libs. By Paul Ramsey<br />
<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Database.com<br />
| [https://multicorn.org/ Multicorn]<br />
| BSD<br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Dun & Badstreet<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
|<br />
|<br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Facebook<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
|<br />
| [https://github.com/hakanensari/frankfurter GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Keycloak<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/schne324/foreign-keycloak-wrapper GitHub]<br />
| [https://pgxn.org/dist/foreign-keycloak-wrapper/ PGXN]<br />
| [https://github.com/schne324/foreign-keycloak-wrapper/blob/master/README.md README]<br />
| Direct database integration with the [https://www.keycloak.org Keycloak] open-source Identity/Access Management solution.<br />
|-<br />
| Mailchimp<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
|<br />
|<br />
| Beta<br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| S3CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
|<br />
|<br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| Telegram<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/telegram_fdw GitHub]<br />
|<br />
|<br />
| telegram_fdw is a Telegram BOT implemented using the PostgreSQL foreign data wrapper interface.<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
|<br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
|<br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [https://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Open Weather Map<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/owmfdw GitHub]<br />
|<br />
|<br />
| A FDW for Open Weather Map (single city)<br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elasticsearch<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/matthewfranglen/postgres-elasticsearch-fdw GitHub]<br />
|<br />
|<br />
| Supports up to PG 13, ES 7.<br />
|-<br />
| Google BigQuery<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
|<br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
|<br />
|<br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://www.openscg.com/bigsql/hadoopfdw/ Bitbucket]<br />
|<br />
|<br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Hive<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
|<br />
|<br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://impala.apache.org/ Impala]<br />
| Native<br />
| BSD<br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://arrow.apache.org/ Apache Arrow]<br />
| Native<br />
| GPLv2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
|<br />
| A part of PG-Strom feature; as a columnar data source with support of SSD-to-GPU Direct SQL <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
|<br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
|<br />
|<br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Log files<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OpenStack / Telemetry<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OS Query<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
|<br />
|<br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
|<br />
|<br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
|<br />
|<br />
| A foreign datawrapper for querying system stats based on [https://libstatgrab.org/ statgrab]<br />
|-<br />
| Environment Variables<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/pgsql-tw/envfdw GitHub]<br />
|<br />
|<br />
| envFDW is a forign data wrapper for processing environment variables<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| faker_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/faker_fdw GitHub]<br />
|<br />
|<br />
| faker_fdw is a foreign data wrapper for PostgreSQL that generates fake data.<br />
|-<br />
| fdw_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
|<br />
|<br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PPG<br />
| Native<br />
|<br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
|<br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www2.meethue.com/en-us/philips-hue-benefits Phillips Hue Lighting Systems]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Random Number<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
|<br />
|<br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| Native<br />
| PostgreSQL<br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
|<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator<br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|-<br />
| VMware vSphere<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/vspherefdw GitHub]<br />
|<br />
|<br />
| A PostgreSQL FDW to query your VMware vSphere service<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
|<br />
|<br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
|<br />
|<br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Black Hole<br />
|<br />
|<br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
|<br />
|<br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [https://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Systemd&diff=36265Systemd2021-07-26T01:57:02Z<p>Barwick: link to relevant documentation section</p>
<hr />
<div>== Service unit configuration ==<br />
<br />
With PostgreSQL 9.6 or newer, it is recommended to build with <code>--with-systemd</code> and use the unit file shown in the [https://www.postgresql.org/docs/current/server-start.html documentation].<br />
<br />
With older releases, the recommended approach is to write a unit file using <code>Type=forking</code> and use <code>pg_ctl</code> for <code>ExecStart</code> and <code>ExecStop</code>.<br />
<br />
== RemoveIPC ==<br />
<br />
The setting<br />
<br />
RemoveIPC=<br />
<br />
in [https://www.freedesktop.org/software/systemd/man/logind.conf.html <code>logind.conf</code>] controls whether System V IPC objects are removed when a user fully logs out. System users are exempt.<br />
<br />
This was turned on by default in systemd version 212 (2014-03-25). RHEL7 ships 219. Debian stable (jessie) ships 215. Apparently, the systemd package in RHEL7 is built with it defaulting to off.<br />
<br />
The observed effect of this is that the semaphore objects used by a PostgreSQL server are removed at apparently random times, leading to the server crashing with log messages like<br />
<br />
LOG: semctl(1234567890, 0, IPC_RMID, ...) failed: Invalid argument<br />
<br />
Shared memory segments that are still attached to are not cleaned up, so systemd will not remove shared memory segments still in use. But semaphores have no concept of a process being attached, so they will be cleaned up even if they are actually still in use.<br />
<br />
A "user logging out" might happen as part of a maintenance job or manually when an administrator logs in as the <code>postgres</code> user or similar, so it is hard to prevent.<br />
<br />
What is a "system user" is determined at systemd compile time from the <code>SYS_UID_MAX</code> setting in <code>/etc/login.defs</code>.<br />
<br />
It is recommended to set<br />
<br />
RemoveIPC=no<br />
<br />
on all server hosts used for PostgreSQL.<br />
<br />
Also, packaging and deployment scripts should be careful to create the <code>postgres</code> user as a system user by using <code>useradd -r</code> or equivalent.<br />
<br />
'''You have to do at least one of these two things, or your PostgreSQL server will be very unreliable.'''<br />
<br />
See also the documentation section [https://www.postgresql.org/docs/current/kernel-resources.html#SYSTEMD-REMOVEIPC systemd RemoveIPC].<br />
<br />
== KillUserProcesses ==<br />
<br />
The setting<br />
<br />
KillUserProcesses=<br />
<br />
in [https://www.freedesktop.org/software/systemd/man/logind.conf.html <code>logind.conf</code>] controls whether all processes of a user's session should be killed when the user logs out of that session.<br />
<br />
This was turned on by default in systemd version 230 (2016-05-21). Currently (August 2016), this is not yet shipped widely (Fedora Branched/25, Debian testing, stable-backports).<br />
<br />
This basically means that if you start a background server process such as PostgreSQL from the shell, it will be killed when you log out, unless you take special care. This has no harmful impact on PostgreSQL server instances started through system service units. It only affects background processes started from a user shell. This could apply if you ''don't'' use systemd service units but instead orchestrate the PostgreSQL server (or any other server process) using traditional init scripts or similar scripts.<br />
<br />
There are various ways to adjust that, including <code>KillOnlyUsers=</code>, <code>KillExcludeUsers=</code>, <code>loginctl enable-linger</code>, <code>systemd-run</code>. These are all explained on the <code>logind.conf</code> man page. Note that being a "system user" has no influence here.</div>Barwickhttps://wiki.postgresql.org/index.php?title=Systemd&diff=36264Systemd2021-07-26T01:54:51Z<p>Barwick: update documentation link (point to /current/ rather than /devel/)</p>
<hr />
<div>== Service unit configuration ==<br />
<br />
With PostgreSQL 9.6 or newer, it is recommended to build with <code>--with-systemd</code> and use the unit file shown in the [https://www.postgresql.org/docs/current/server-start.html documentation].<br />
<br />
With older releases, the recommended approach is to write a unit file using <code>Type=forking</code> and use <code>pg_ctl</code> for <code>ExecStart</code> and <code>ExecStop</code>.<br />
<br />
== RemoveIPC ==<br />
<br />
The setting<br />
<br />
RemoveIPC=<br />
<br />
in [https://www.freedesktop.org/software/systemd/man/logind.conf.html <code>logind.conf</code>] controls whether System V IPC objects are removed when a user fully logs out. System users are exempt.<br />
<br />
This was turned on by default in systemd version 212 (2014-03-25). RHEL7 ships 219. Debian stable (jessie) ships 215. Apparently, the systemd package in RHEL7 is built with it defaulting to off.<br />
<br />
The observed effect of this is that the semaphore objects used by a PostgreSQL server are removed at apparently random times, leading to the server crashing with log messages like<br />
<br />
LOG: semctl(1234567890, 0, IPC_RMID, ...) failed: Invalid argument<br />
<br />
Shared memory segments that are still attached to are not cleaned up, so systemd will not remove shared memory segments still in use. But semaphores have no concept of a process being attached, so they will be cleaned up even if they are actually still in use.<br />
<br />
A "user logging out" might happen as part of a maintenance job or manually when an administrator logs in as the <code>postgres</code> user or similar, so it is hard to prevent.<br />
<br />
What is a "system user" is determined at systemd compile time from the <code>SYS_UID_MAX</code> setting in <code>/etc/login.defs</code>.<br />
<br />
It is recommended to set<br />
<br />
RemoveIPC=no<br />
<br />
on all server hosts used for PostgreSQL.<br />
<br />
Also, packaging and deployment scripts should be careful to create the <code>postgres</code> user as a system user by using <code>useradd -r</code> or equivalent.<br />
<br />
'''You have to do at least one of these two things, or your PostgreSQL server will be very unreliable.'''<br />
<br />
== KillUserProcesses ==<br />
<br />
The setting<br />
<br />
KillUserProcesses=<br />
<br />
in [https://www.freedesktop.org/software/systemd/man/logind.conf.html <code>logind.conf</code>] controls whether all processes of a user's session should be killed when the user logs out of that session.<br />
<br />
This was turned on by default in systemd version 230 (2016-05-21). Currently (August 2016), this is not yet shipped widely (Fedora Branched/25, Debian testing, stable-backports).<br />
<br />
This basically means that if you start a background server process such as PostgreSQL from the shell, it will be killed when you log out, unless you take special care. This has no harmful impact on PostgreSQL server instances started through system service units. It only affects background processes started from a user shell. This could apply if you ''don't'' use systemd service units but instead orchestrate the PostgreSQL server (or any other server process) using traditional init scripts or similar scripts.<br />
<br />
There are various ways to adjust that, including <code>KillOnlyUsers=</code>, <code>KillExcludeUsers=</code>, <code>loginctl enable-linger</code>, <code>systemd-run</code>. These are all explained on the <code>logind.conf</code> man page. Note that being a "system user" has no influence here.</div>Barwickhttps://wiki.postgresql.org/index.php?title=Greatest_Common_Divisor&diff=36098Greatest Common Divisor2021-06-02T01:36:07Z<p>Barwick: Note availability of gcd() function from PostgreSQL 13.</p>
<hr />
<div>{{SnippetInfo|Greatest Common Divison|version=8.4|lang=SQL|category=Library}}<br />
<br />
Using a CTE, the calculation of greatest common divisor (gcd) is trivial.<br />
<br />
<source lang="SQL"><br />
CREATE OR REPLACE FUNCTION gcd( a bigint, b bigint)<br />
RETURNS bigint<br />
IMMUTABLE<br />
STRICT<br />
LANGUAGE SQL<br />
AS $$<br />
WITH RECURSIVE t(a,b) AS (<br />
VALUES (abs($1)::bigint, abs($2)::bigint)<br />
UNION ALL<br />
SELECT b, mod(a,b) FROM t<br />
WHERE b > 0<br />
)<br />
SELECT a FROM t WHERE b = 0<br />
$$;<br />
</source><br />
<br />
Thanks to David Fetter for this query.<br />
Thanks to Pasha Golub for the faster, more correct version.<br />
<br />
From PostgreSQL 13, a built-in function [https://pgpedia.info/g/gcd-function.html gcd()] is available (PostgreSQL documentation: [https://www.postgresql.org/docs/current/functions-math.html#FUNCTIONS-MATH-FUNC-TABLE Mathematical Functions ]).</div>Barwickhttps://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=35894Foreign data wrappers2021-04-09T23:17:15Z<p>Barwick: /* Specific SQL Database Wrappers */ remove obsolete SQLite FDW implementation (see comment in its GitHub README)</p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained ?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| [https://gdal.org/drivers/vector/index.html GDAL/OGR]<br />
| Native<br />
| MIT<br />
| [https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
| yum.postgresql.org, apt.postgresql.org, and part of PostGIS windows bundle (application stackbuilder)<br />
| <br />
| Can access many kinds of data sources (Relational databases, spreadsheets, CSV files, web feature services, etc). Uses the [https://gdal.org/ GDAL library] which supports hundreds of formats to access the data. Exposes vector data as PostGIS geometry columns if you have PostGIS installed. Works great with both spatial and non-spatial data.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|}<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/postgres-fdw.html documentation]<br />
|<br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
|<br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|FDW for MySQL<br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.firebirdsql.org/ Firebird]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|[https://github.com/ibarwick/firebird_fdw/blob/master/README.md README]<br />
|version [https://github.com/ibarwick/firebird_fdw/releases/tag/1.2.0 1.2.0] released (2020-10)<br />
|-<br />
|[https://www.sqlite.org/index.html SQLite]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pgspider/sqlite_fdw github]<br />
|[https://pgxn.org/dist/sqlite_fdw PGXN]<br />
|[https://github.com/pgspider/sqlite_fdw/blob/master/README.md README]<br />
|An FDW for SQLite3 (write support and several pushdown optimization)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[https://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[https://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|Native<br />
|Apache<br />
|[https://github.com/adjust/clickhouse_fdw Github]<br />
|<br />
|[https://github.com/adjust/clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
| [https://github.com/griddb/griddb_nosql GridDB]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/griddb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/griddb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| InfluxDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/influxdb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/influxdb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| [https://kafka.apache.org/ Kafka]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/kafka_fdw GitHub]<br />
|<br />
| [https://github.com/adjust/kafka_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|[https://multicorn.org/ Multicorn]<br />
|GPLv3<br />
|[https://github.com/sim51/neo4j-fdw Github]<br />
|<br />
|[https://github.com/sim51/neo4j-fdw/blob/master/README.adoc README]<br />
|FWD for Neo4j and also add a Cypher function to Pg<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/slamdata/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://github.com/facebook/rocksdb RocksDB]<br />
|Native<br />
|Apache<br />
|[https://github.com/vidardb/PostgresForeignDataWrapper Github]<br />
|<br />
|[https://github.com/vidardb/PostgresForeignDataWrapper/blob/master/README.md README]<br />
|FDW for RocksDB<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to]<br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| CSV / gzipped<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
|<br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
|<br />
|-<br />
| Multi-File<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| Parquet<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/parquet_fdw GitHub]<br />
|<br />
|<br />
| Foreign data wrapper for reading Parquet files using libarrow/libparquet<br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| XML<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [https://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [https://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [https://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
|<br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
|<br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
|<br />
|}<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ICAL<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
|<br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
|<br />
|-<br />
| IMAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
|<br />
|-<br />
| RSS<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed.<br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|-<br />
| pgsql-http<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pramsey/pgsql-http GitHub]<br />
| Compile<br />
| <br />
| Allows to query any http resource using CURL libs. By Paul Ramsey<br />
<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Database.com<br />
| [https://multicorn.org/ Multicorn]<br />
| BSD<br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Dun & Badstreet<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
|<br />
|<br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Facebook<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
|<br />
| [https://github.com/hakanensari/frankfurter GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Keycloak<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/schne324/foreign-keycloak-wrapper GitHub]<br />
| [https://pgxn.org/dist/foreign-keycloak-wrapper/ PGXN]<br />
| [https://github.com/schne324/foreign-keycloak-wrapper/blob/master/README.md README]<br />
| Direct database integration with the [https://www.keycloak.org Keycloak] open-source Identity/Access Management solution.<br />
|-<br />
| Mailchimp<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
|<br />
|<br />
| Beta<br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| S3CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
|<br />
|<br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| Telegram<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/telegram_fdw GitHub]<br />
|<br />
|<br />
| telegram_fdw is a Telegram BOT implemented using the PostgreSQL foreign data wrapper interface.<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
|<br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
|<br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [https://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Open Weather Map<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/owmfdw GitHub]<br />
|<br />
|<br />
| A FDW for Open Weather Map (single city)<br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elasticsearch<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/matthewfranglen/postgres-elasticsearch-fdw GitHub]<br />
|<br />
|<br />
| Supports up to PG 13, ES 7.<br />
|-<br />
| Google BigQuery<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
|<br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
|<br />
|<br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://www.openscg.com/bigsql/hadoopfdw/ Bitbucket]<br />
|<br />
|<br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Hive<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
|<br />
|<br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://impala.apache.org/ Impala]<br />
| Native<br />
| BSD<br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://arrow.apache.org/ Apache Arrow]<br />
| Native<br />
| GPLv2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
|<br />
| A part of PG-Strom feature; as a columnar data source with support of SSD-to-GPU Direct SQL <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
|<br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
|<br />
|<br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Log files<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OpenStack / Telemetry<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OS Query<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
|<br />
|<br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
|<br />
|<br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
|<br />
|<br />
| A foreign datawrapper for querying system stats based on [https://libstatgrab.org/ statgrab]<br />
|-<br />
| Environment Variables<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/pgsql-tw/envfdw GitHub]<br />
|<br />
|<br />
| envFDW is a forign data wrapper for processing environment variables<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| faker_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/faker_fdw GitHub]<br />
|<br />
|<br />
| faker_fdw is a foreign data wrapper for PostgreSQL that generates fake data.<br />
|-<br />
| fdw_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
|<br />
|<br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PPG<br />
| Native<br />
|<br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
|<br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www2.meethue.com/en-us/philips-hue-benefits Phillips Hue Lighting Systems]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Random Number<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
|<br />
|<br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| [https://multicorn.org/ Multicorn]<br />
| Native<br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
| PostgreSQL<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator<br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|-<br />
| VMware vSphere<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/vspherefdw GitHub]<br />
|<br />
|<br />
| A PostgreSQL FDW to query your VMware vSphere service<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
|<br />
|<br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
|<br />
|<br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Black Hole<br />
|<br />
|<br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
|<br />
|<br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [https://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Psqlrc&diff=35578Psqlrc2020-12-21T01:39:39Z<p>Barwick: remove section containing a sole invalid line</p>
<hr />
<div>psqlrc is the configuration file read by the [[https://www.postgresql.org/docs/current/static/app-psql.html psql client]]. psql will typically read both a systemwide psqlrc and the per-user one (in ~/.psqlrc on unix-ish systems)<br />
<br />
It allows a lot of customization of psql, including defining macros to implement new commands.<br />
<br />
TODO: Needs curation. This is a work in progress, collecting existing examples of psqlrc configuration, in the hope someone will go through and pull out some interesting snippets, and maybe a kitchen sink psqlrc that has ''all the good bits'' in it.<br />
<br />
== Articles ==<br />
<br />
* https://robots.thoughtbot.com/an-explained-psqlrc<br />
* https://www.digitalocean.com/community/tutorials/how-to-customize-the-postgresql-prompt-with-psqlrc-on-ubuntu-14-04<br />
* https://opensourcedbms.com/dbms/psqlrc-psql-startup-file-for-postgres/<br />
* https://www.if-not-true-then-false.com/2009/postgresql-psql-psqlrc-tips-and-tricks/<br />
* https://www.citusdata.com/blog/2017/07/16/customizing-my-postgres-shell-using-psqlrc/<br />
* http://pavdmyt.com/better-postgresql-cli-experience-with-psqlrc-tweaks/<br />
* http://www.craigkerstiens.com/2013/02/21/more-out-of-psql/<br />
* http://merlinmoncure.blogspot.com/2012/09/psql-now-with-splash-of-color.html<br />
* http://i-dba.blogspot.com/2014/02/colorizing-psql-prompt-guide.html<br />
* https://simply.name/yet-another-psql-color-prompt.html<br />
* https://www.periscopedata.com/blog/optimizing-your-psql.html<br />
* https://www.depesz.com/2008/05/18/silencing-commands-in-psqlrc/<br />
* https://www.commandprompt.com/blog/psql_tip_change_history_location/<br />
<br />
<br />
== Examples ==<br />
<br />
* https://bitbucket.org/adamkg/libakg/src/default/dot (clever - check out [https://bitbucket.org/adamkg/libakg/src/default/dot/psqlrc psqlrc] and [https://bitbucket.org/adamkg/libakg/src/default/dot/psqlrc-commands.d/update_psqlrc_commands.sh psqlrc-commands.d/update_psqlrc_commands.sh]<br />
* https://github.com/e7e6/psqlrc/blob/master/psqlrc<br />
* https://github.com/aziz/dotfiles/blob/master/psqlrc</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=35569Repmgr2020-12-09T01:07:37Z<p>Barwick: And update the displayed version number too...</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL replication and failover management, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements. It helps DBAs and System Administrators manage a cluster of PostgreSQL databases by taking advantage of the Hot Standby capability introduced in PostgreSQL 9.<br />
<br />
repmgr streamlines the process of administration and daily management, enhances productivity, complements built-in replication capabilities, and reduces overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process.<br />
* providing support for HA operations such as switchovers and failovers.<br />
<br />
repmgr is available via the 2ndQuadrant package repositories as well as the PGDG community repositories. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 5 is compatible with PostgreSQL 9.4 and above. See the [https://repmgr.org/docs/current/install-requirements.html#INSTALL-COMPATIBILITY-MATRIX repmgr compatibility matrix] for full details.<br />
<br />
<br />
== repmgr 5 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/current/release-5.2.1.html 5.2.1] (2020-12-08).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or Barman<br />
* Timeline following:<br />
** a standby that can be promoted to a primary without requiring a restart<br />
** other standbys that can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the primary to another standby mode<br />
* Replication slot support , simplifying WAL retention management<br />
* [https://repmgr.org/docs/current/performing-switchover.html Switchover] support for role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Package installation instructions for Red Hat/Debian-based distributions can be found [https://repmgr.org/docs/current/installation-packages.html here].<br />
<br />
== Documentation ==<br />
<br />
* [https://repmgr.org/docs/current/index.html repmgr documentation]<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=35568Repmgr2020-12-09T01:07:10Z<p>Barwick: Add latest release</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL replication and failover management, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements. It helps DBAs and System Administrators manage a cluster of PostgreSQL databases by taking advantage of the Hot Standby capability introduced in PostgreSQL 9.<br />
<br />
repmgr streamlines the process of administration and daily management, enhances productivity, complements built-in replication capabilities, and reduces overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process.<br />
* providing support for HA operations such as switchovers and failovers.<br />
<br />
repmgr is available via the 2ndQuadrant package repositories as well as the PGDG community repositories. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 5 is compatible with PostgreSQL 9.4 and above. See the [https://repmgr.org/docs/current/install-requirements.html#INSTALL-COMPATIBILITY-MATRIX repmgr compatibility matrix] for full details.<br />
<br />
<br />
== repmgr 5 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/current/release-5.2.1.html 5.2] (2020-12-08).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or Barman<br />
* Timeline following:<br />
** a standby that can be promoted to a primary without requiring a restart<br />
** other standbys that can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the primary to another standby mode<br />
* Replication slot support , simplifying WAL retention management<br />
* [https://repmgr.org/docs/current/performing-switchover.html Switchover] support for role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Package installation instructions for Red Hat/Debian-based distributions can be found [https://repmgr.org/docs/current/installation-packages.html here].<br />
<br />
== Documentation ==<br />
<br />
* [https://repmgr.org/docs/current/index.html repmgr documentation]<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=35474Repmgr2020-10-22T12:08:26Z<p>Barwick: note repmgr 5.2 release</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL replication and failover management, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements. It helps DBAs and System Administrators manage a cluster of PostgreSQL databases by taking advantage of the Hot Standby capability introduced in PostgreSQL 9.<br />
<br />
repmgr streamlines the process of administration and daily management, enhances productivity, complements built-in replication capabilities, and reduces overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process.<br />
* providing support for HA operations such as switchovers and failovers.<br />
<br />
repmgr is available via the 2ndQuadrant package repositories as well as the PGDG community repositories. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 5 is compatible with PostgreSQL 9.4 and above. See the [https://repmgr.org/docs/current/install-requirements.html#INSTALL-COMPATIBILITY-MATRIX repmgr compatibility matrix] for full details.<br />
<br />
<br />
== repmgr 5 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/current/release-5.2.0.html 5.2] (2020-10-22).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or Barman<br />
* Timeline following:<br />
** a standby that can be promoted to a primary without requiring a restart<br />
** other standbys that can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the primary to another standby mode<br />
* Replication slot support , simplifying WAL retention management<br />
* [https://repmgr.org/docs/current/performing-switchover.html Switchover] support for role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Package installation instructions for Red Hat/Debian-based distributions can be found [https://repmgr.org/docs/current/installation-packages.html here].<br />
<br />
== Documentation ==<br />
<br />
* [https://repmgr.org/docs/current/index.html repmgr documentation]<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=User:Barwick&diff=35471User:Barwick2020-10-19T01:33:11Z<p>Barwick: Update personal info</p>
<hr />
<div>Ian Barwick. DBA and developer based in Tokyo, Japan. PostgreSQL user since 2001 (version 7.1). Currently working for EnterpriseDB.<br />
<br />
== Maintainer of ==<br />
<br />
* [https://repmgr.org/ repmgr]<br />
* [https://github.com/ibarwick/firebird_fdw/ firebird_fdw]<br />
<br />
== Links ==<br />
<br />
* [https://sql-info.de/postgresql/notes/index.html My PostgreSQL blog]<br />
* [https://github.com/ibarwick/ GitHub]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=35470Foreign data wrappers2020-10-19T01:29:09Z<p>Barwick: Update firebird_fdw info</p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained ?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| [https://gdal.org/drivers/vector/index.html GDAL/OGR]<br />
| Native<br />
| MIT<br />
| [https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
| yum.postgresql.org, apt.postgresql.org, and part of PostGIS windows bundle (application stackbuilder)<br />
| <br />
| Can access many kinds of data sources (Relational databases, spreadsheets, CSV files, web feature services, etc). Uses the [https://gdal.org/ GDAL library] which supports hundreds of formats to access the data. Exposes vector data as PostGIS geometry columns if you have PostGIS installed. Works great with both spatial and non-spatial data.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|}<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/postgres-fdw.html documentation]<br />
|<br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
|<br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|FDW for MySQL<br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.firebirdsql.org/ Firebird]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|[https://github.com/ibarwick/firebird_fdw/blob/master/README.md README]<br />
|version [https://github.com/ibarwick/firebird_fdw/releases/tag/1.2.0 1.2.0] released (2020-10)<br />
|-<br />
|SQLite<br />
|Native<br />
|<br />
|[https://github.com/gleu/sqlite_fdw github]<br />
|<br />
|<br />
|An FDW for SQLite3 (read-only)<br />
|-<br />
|[https://www.sqlite.org/index.html SQLite]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pgspider/sqlite_fdw github]<br />
|[https://pgxn.org/dist/sqlite_fdw PGXN]<br />
|[https://github.com/pgspider/sqlite_fdw/blob/master/README.md README]<br />
|An FDW for SQLite3 (write support and several pushdown optimization)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[https://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[https://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|Native<br />
|Apache<br />
|[https://github.com/adjust/clickhouse_fdw Github]<br />
|<br />
|[https://github.com/adjust/clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
| [https://github.com/griddb/griddb_nosql GridDB]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/griddb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/griddb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| InfluxDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/influxdb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/influxdb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| [https://kafka.apache.org/ Kafka]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/kafka_fdw GitHub]<br />
|<br />
| [https://github.com/adjust/kafka_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|[https://multicorn.org/ Multicorn]<br />
|GPLv3<br />
|[https://github.com/sim51/neo4j-fdw Github]<br />
|<br />
|[https://github.com/sim51/neo4j-fdw/blob/master/README.adoc README]<br />
|FWD for Neo4j and also add a Cypher function to Pg<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/slamdata/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://github.com/facebook/rocksdb RocksDB]<br />
|Native<br />
|Apache<br />
|[https://github.com/vidardb/PostgresForeignDataWrapper Github]<br />
|<br />
|[https://github.com/vidardb/PostgresForeignDataWrapper/blob/master/README.md README]<br />
|FDW for RocksDB<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to]<br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| CSV / gzipped<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
|<br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
|<br />
|-<br />
| Multi-File<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| Parquet<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/adjust/parquet_fdw GitHub]<br />
|<br />
|<br />
| Foreign data wrapper for reading Parquet files using libarrow/libparquet<br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| XML<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [https://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [https://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [https://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
|<br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
|<br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
|<br />
|}<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ICAL<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
|<br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
|<br />
|-<br />
| IMAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
|<br />
|-<br />
| RSS<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed.<br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|-<br />
| pgsql-http<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pramsey/pgsql-http GitHub]<br />
| Compile<br />
| <br />
| Allows to query any http resource using CURL libs. By Paul Ramsey<br />
<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Database.com<br />
| [https://multicorn.org/ Multicorn]<br />
| BSD<br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Dun & Badstreet<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
|<br />
|<br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Facebook<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
|<br />
| [https://github.com/hakanensari/frankfurter GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Keycloak<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/schne324/foreign-keycloak-wrapper GitHub]<br />
| [https://pgxn.org/dist/foreign-keycloak-wrapper/ PGXN]<br />
| [https://github.com/schne324/foreign-keycloak-wrapper/blob/master/README.md README]<br />
| Direct database integration with the [https://www.keycloak.org Keycloak] open-source Identity/Access Management solution.<br />
|-<br />
| Mailchimp<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
|<br />
|<br />
| Beta<br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| S3CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
|<br />
|<br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| Telegram<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/telegram_fdw GitHub]<br />
|<br />
|<br />
| telegram_fdw is a Telegram BOT implemented using the PostgreSQL foreign data wrapper interface.<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
|<br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
|<br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [https://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Open Weather Map<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/owmfdw GitHub]<br />
|<br />
|<br />
| A FDW for Open Weather Map (single city)<br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elasticsearch<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/matthewfranglen/postgres-elasticsearch-fdw GitHub]<br />
|<br />
|<br />
| Supports up to PG 12, ES 7.<br />
|-<br />
| Google BigQuery<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
|<br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
|<br />
|<br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://www.openscg.com/bigsql/hadoopfdw/ Bitbucket]<br />
|<br />
|<br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Hive<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
|<br />
|<br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://impala.apache.org/ Impala]<br />
| Native<br />
| BSD<br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://arrow.apache.org/ Apache Arrow]<br />
| Native<br />
| GPLv2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
|<br />
| A part of PG-Strom feature; as a columnar data source with support of SSD-to-GPU Direct SQL <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
|<br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
|<br />
|<br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Log files<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OpenStack / Telemetry<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OS Query<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
|<br />
|<br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
|<br />
|<br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
|<br />
|<br />
| A foreign datawrapper for querying system stats based on [https://libstatgrab.org/ statgrab]<br />
|-<br />
| Environment Variables<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/pgsql-tw/envfdw GitHub]<br />
|<br />
|<br />
| envFDW is a forign data wrapper for processing environment variables<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| faker_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/faker_fdw GitHub]<br />
|<br />
|<br />
| faker_fdw is a foreign data wrapper for PostgreSQL that generates fake data.<br />
|-<br />
| fdw_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
|<br />
|<br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PPG<br />
| Native<br />
|<br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
|<br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www2.meethue.com/en-us/philips-hue-benefits Phillips Hue Lighting Systems]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Random Number<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
|<br />
|<br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| [https://multicorn.org/ Multicorn]<br />
| Native<br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
| PostgreSQL<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator<br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|-<br />
| VMware vSphere<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/ycku/vspherefdw GitHub]<br />
|<br />
|<br />
| A PostgreSQL FDW to query your VMware vSphere service<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
|<br />
|<br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
|<br />
|<br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Black Hole<br />
|<br />
|<br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
|<br />
|<br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [https://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Tuning_Your_PostgreSQL_Server&diff=35462Tuning Your PostgreSQL Server2020-10-14T11:57:18Z<p>Barwick: Update pgBadger links (and remove dead PgFouine link)</p>
<hr />
<div>__NOTOC__<br />
''by Greg Smith, Robert Treat, and Christopher Browne''<br />
<br />
{{Languages}}<br />
<br />
PostgreSQL ships with a basic configuration tuned for wide compatibility rather than performance. Odds are good the default parameters are very undersized for your system. Rather than get dragged into the details of everything you should eventually know (which you can find if you want it at the [http://www.pgcon.org/2008/schedule/events/104.en.html GUC Three Hour Tour]), here we're going to sprint through a simplified view of the basics, with a look at the most common things people new to PostgreSQL aren't aware of. You should click on the name of the parameter in each section to jump to the relevant documentation in the PostgreSQL manual for more details after reading the quick intro here. There is also additional information available about many of these parameters, as well as a list of parameters you shouldn't adjust, at [https://www.packtpub.com/article/server-configuration-tuning-postgresql Server Configuration Tuning].<br />
<br />
== Background Information on Configuration Settings ==<br />
<br />
PostgreSQL settings can be manipulated a number of different ways, but generally you will want them changed in your configuration files, either directly or, starting with PostgreSQL 9.4, through [http://www.postgresql.org/docs/current/static/sql-altersystem.html <tt>ALTER SYSTEM</tt>]. The specific options available change from release to release, the definitive list is in the source code at src/backend/utils/misc/guc.c for your version of PostgreSQL (but the pg_settings view works well enough for most purposes).<br />
<br />
=== The types of settings ===<br />
<br />
There are several different types of configuration settings, divided up based on the possible inputs they take<br />
<br />
* Boolean: true, false, on, off<br />
* Integer: Whole numbers (2112)<br />
* Float: Decimal values (21.12)<br />
* Memory / Disk: Integers (2112) or "computer units" (512MB, 2112GB). Avoid integers--you need to know the underlying unit to figure out what they mean.<br />
* Time: "Time units" aka d,m,s (30s). Sometimes the unit is left out; don't do that<br />
* Strings: Single quoted text ('pg_log')<br />
* ENUMs: Strings, but from a specific list ('WARNING', 'ERROR')<br />
* Lists: A comma separated list of strings ('"$user",public,tsearch2) <br />
<br />
=== When they take effect ===<br />
<br />
PostgreSQL settings have different levels of flexibility for when they can be changed, usually related to internal code restrictions. The complete list of levels is:<br />
<br />
* Postmaster: requires restart of server <br />
* Sighup: requires a HUP of the server, either by kill -HUP (usually -1), pg_ctl reload, or <tt>SELECT pg_reload_conf()</tt>;<br />
* User: can be set within individual sessions, take effect only within that session<br />
* Internal: set at compile time, can't be changed, mainly for reference<br />
* Backend: settings which must be set before session start<br />
* Superuser: can be set at runtime for the server by superusers<br />
<br />
Most of the time you'll only use the first of these, but the second can be useful if you have a server you don't want to take down, while the user session settings can be helpful for some special situations. You can tell which type of parameter a setting is by looking at the "context" field in the pg_settings view.<br />
<br />
=== Important notes about configuration files ===<br />
<br />
* Command line options override postgresql.auto.conf settings override postgresql.conf settings.<br />
* If the same setting is listed multiple times, the last one wins.<br />
* You can figure out the postgresql.conf location with <tt>SHOW config_file</tt>. It will generally be $PGDATA/postgresql.conf (<tt>SHOW data_directory</tt>), but watch out for symbolic links, [http://www.postgresql.org/docs/current/static/app-pg-ctl.html#AEN93617 postmaster.opts] and other trickiness<br />
* Lines with # are comments and have no effect. For a new database, this will mean the setting is using the default, but on running systems this may not hold true! Changes to the configuration files do not take effect without a reload/restart, so it's possible for the system to be running something different from what is in the file. <br />
<br />
=== Viewing the current settings === <br />
<br />
* Look at the configuration files. This is generally not definitive!<br />
* <tt>SHOW ALL</tt>, <tt>SHOW <setting></tt> will show you the current value of the setting. Watch out for session specific changes<br />
* <tt>SELECT * FROM pg_settings</tt> will label session specific changes as locally modified<br />
<br />
=== Tuning tools ===<br />
<br />
* [https://www.devart.com/dbforge/postgresql/studio/query-profiler.html dbForge Studio for PostgreSQL] helps to identify productivity bottlenecks, and provides PostgreSQL performance tuning.<br />
* The [https://github.com/jfcoz/postgresqltuner postgresqltuner.pl] script can analyze the configuration and make tuning recommendations. <br />
* [https://pgbadger.darold.net/ PgBadger] analyse PostgreSQL logs to generate performance reports.<br />
* [https://www.pgmustard.com/ pgMustard] provides tuning advice based on EXPLAIN ANALYZE output.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-LISTEN-ADDRESSES listen_addresses] ==<br />
<br />
By default, PostgreSQL only responds to connections from the local host. If you want your server to be accessible from other systems via standard TCP/IP networking, you need to change listen_addresses from its default. The usual approach is to set it to listen to all addresses like this:<br />
<br />
<code><pre><br />
listen_addresses = '*'<br />
</pre></code><br />
<br />
And then control who can and cannot connect via the [http://www.postgresql.org/docs/current/static/auth-pg-hba-conf.html pg_hba.conf] file.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-MAX-CONNECTIONS max_connections]==<br />
<br />
max_connections sets exactly that: the maximum number of client connections allowed. This is very important to some of the below parameters (particularly work_mem) because there are some memory resources that are or can be allocated on a per-client basis, so the maximum number of clients suggests the maximum possible memory use. Generally, PostgreSQL on good hardware can support a few hundred connections. If you want to have thousands instead, you should consider using [[Replication, Clustering, and Connection Pooling|connection pooling software]] to reduce the connection overhead.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-SHARED-BUFFERS shared_buffers]==<br />
<br />
The shared_buffers configuration parameter determines how much memory is dedicated to PostgreSQL to use for caching data. One reason the defaults are low is because on some platforms (like older Solaris versions and SGI), having large values requires invasive action like recompiling the kernel. Even on a modern Linux system, the stock kernel will likely not allow setting shared_buffers to over 32MB without adjusting kernel settings first. (PostgreSQL 9.4 and later use a different shared memory mechanism, so kernel settings will usually not have to be adjusted there.)<br />
<br />
If you have a system with 1GB or more of RAM, a reasonable starting value for shared_buffers is 1/4 of the memory in your system. If you have less RAM you'll have to account more carefully for how much RAM the OS is taking up; closer to 15% is more typical there. There are some workloads where even larger settings for shared_buffers are effective, but given the way PostgreSQL also relies on the operating system cache, it's unlikely you'll find using more than 40% of RAM to work better than a smaller amount.<br />
<br />
Be aware that if your system or PostgreSQL build is 32-bit, it might not be practical to set shared_buffers above 2 ~ 2.5GB. See [http://rhaas.blogspot.jp/2011/05/sharedbuffers-on-32-bit-systems.html this blog post] for details.<br />
<br />
Note that on Windows, large values for shared_buffers aren't as effective, and you may find better results keeping it relatively low and using the OS cache more instead. On Windows the useful range is 64MB to 512MB.<br />
<br />
Changing this setting requires restarting the database. Also, this is a hard allocation of memory; the whole thing gets allocated out of virtual memory when the database starts.<br />
<br />
;PostgreSQL 9.2 or earlier<br />
<br />
If you are running PostgreSQL 9.2 or earlier, it's likely that in order to increase the value of shared_buffers you will have to increase the amount of memory your operating system allows you to allocate to a single shared memory segment. On UNIX-like systems, if you set it above what's supported, you'll get a message like this:<br />
<br />
<code><pre><br />
IpcMemoryCreate: shmget(key=5432001, size=415776768, 03600) failed: Invalid argument <br />
<br />
This error usually means that PostgreSQL's request for a shared memory <br />
segment exceeded your kernel's SHMMAX parameter. You can either <br />
reduce the request size or reconfigure the kernel with larger SHMMAX. <br />
To reduce the request size (currently 415776768 bytes), reduce <br />
PostgreSQL's shared_buffers parameter (currently 50000) and/or <br />
its max_connections parameter (currently 12).<br />
</pre></code><br />
<br />
See [http://www.postgresql.org/docs/current/static/kernel-resources.html Managing Kernel Resources] for details on how to correct this.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-query.html#GUC-EFFECTIVE-CACHE-SIZE effective_cache_size]==<br />
<br />
effective_cache_size should be set to an estimate of how much memory is available for disk caching by the operating system and within the database itself, after taking into account what's used by the OS itself and other applications. This is a guideline for how much memory you expect to be available in the OS and PostgreSQL buffer caches, not an allocation! This value is used only by the PostgreSQL query planner to figure out whether plans it's considering would be expected to fit in RAM or not. If it's set too low, indexes may not be used for executing queries the way you'd expect. The setting for shared_buffers is not taken into account here--only the effective_cache_size value is, so it should include memory dedicated to the database too.<br />
<br />
Setting effective_cache_size to 1/2 of total memory would be a normal conservative setting, and 3/4 of memory is a more aggressive but still reasonable amount. You might find a better estimate by looking at your operating system's statistics. On UNIX-like systems, add the free+cached numbers from free or top to get an estimate. On Windows see the "System Cache" size in the Windows Task Manager's Performance tab. Changing this setting does not require restarting the database (HUP is enough).<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-wal.html#RUNTIME-CONFIG-WAL-CHECKPOINTS checkpoint_segments checkpoint_completion_target]==<br />
<br />
'''Note''': This applies to PostgreSQL 9.4 and below. [https://www.postgresql.org/docs/9.5/release-9-5.html PostgreSQL 9.5 introduced min_wal_size and max_wal_size] configuration parameters and removed checkpoint_segments. Please review the release notes and the documentation on [https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-MIN-WAL-SIZE min_wal_size], [https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-MAX-WAL-SIZE max_wal_size] and [https://www.postgresql.org/docs/current/wal-configuration.html WAL configuration].<br />
<br />
PostgreSQL writes new transactions to the database in files called WAL segments that are 16MB in size. Every time checkpoint_segments worth of these files have been written, by default 3, a checkpoint occurs. Checkpoints can be resource intensive, and on a modern system doing one every 48MB will be a serious performance bottleneck. Setting checkpoint_segments to a much larger value improves that. Unless you're running on a very small configuration, you'll almost certainly be better setting this to at least 10, which also allows usefully increasing the completion target.<br />
<br />
For more write-heavy systems, values from 32 (checkpoint every 512MB) to 256 (every 4GB) are popular nowadays. Very large settings use a lot more disk and will cause your database to take longer to recover, so make sure you're comfortable with both those things before large increases. Normally the large settings (>64/1GB) are only used for bulk loading. Note that whatever you choose for the segments, you'll still get a checkpoint at least every 5 minutes unless you also increase checkpoint_timeout (which isn't necessary on most systems).<br />
<br />
Checkpoint writes are spread out a bit while the system starts working toward the next checkpoint. You can spread those writes out further, lowering the average write overhead, by increasing the checkpoint_completion_target parameter to its useful maximum of 0.9 (aim to finish by the time 90% of the next checkpoint is here) rather than the default of 0.5 (aim to finish when the next one is 50% done). A setting of 0 gives something similar to the behavior of obsolete versions. The main reason the default isn't just 0.9 is that you need a larger checkpoint_segments value than the default for broader spreading to work well. For lots more information on checkpoint tuning, see [http://www.westnet.com/~gsmith/content/postgresql/chkp-bgw-83.htm Checkpoints and the Background Writer] (where you'll also learn why tuning the background writer parameters is challenging to do usefully).<br />
<br />
==[http://www.postgresql.org/docs/current/static/routine-vacuuming.html#AUTOVACUUM autovacuum]==<br />
<br />
The autovacuum process takes care of several maintenance chores inside your database that you really need. Generally, if you think you need to turn regular vacuuming off because it's taking too much time or resources, that means you're doing it wrong. The answer to almost all vacuuming problems is to vacuum more often, not less, so that each individual vacuum operation has less to clean up.<br />
<br />
However, it's acceptable to disable autovacuum for short periods of time, for instance when bulk loading large amounts of data.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-logging.html Logging]==<br />
<br />
There are many things you can log that may or may not be important to you. You should investigate the documentation on all of the options, but here are some tips & tricks to get you started:<br />
<br />
*log_destination & log_directory (& log_filename): What you set these options to is not as important as knowing they can give you hints to determine where your database server is logging to. Best practice would be to try and make this as similar as possible across your servers. Note that in some cases, the init script starting your database may be customizing the log destination in the command line used to start the database, overriding what's in the configuration files (and making it so you'll get different behavior if you run pg_ctl manually instead of using the init script).<br />
<br />
*log_min_error_statement: You should probably make sure this is at least on error, so that you will see any SQL commands which cause an error. should be the default on recent versions. <br />
<br />
*log_min_duration_statement: Not necessary for everyday use, but this can generate [[Logging Difficult Queries|logs of "slow queries"]] on your system. <br />
<br />
*log_line_prefix: Appends information to the start of each line. A good generic recommendation is '%t:%r:%u@%d:[%p]: ' : %t=timestamp, %u=db user name, %r=host connecting from, %d=database connecting to, %p=PID of connection. It may not be obvious what the PID is useful at first, but it can be vital for trying to troubleshoot problems in the future so better to put in the logs from the start.<br />
<br />
*log_statement: Choices of none, ddl, mod, all. Using all or mod in production would introduce overhead of the logging. The performance penalties for "all" would be significant if the workload is select intensive and less significant for write-intensive workloads. Turning the synchronous_commit to off would cause a more severe performance regression. DDL can sometime be helpful to discover rogue changes made outside of your recommend processes, by "cowboy DBAs" for example.<br />
<br />
There are also external tools such [https://pgbadger.darold.net/ pgbadger] that can analyze Postgres logs, see [[Monitoring]] for a comprehensive list.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-query.html#GUC-DEFAULT-STATISTICS-TARGET default_statistics_target]==<br />
<br />
The database software collects statistics about each of the tables in your database to decide how to execute queries against it. If you're not getting good execution query plans particularly on larger (or more varied) tables you should increase default_statistics_target then ANALYZE the database again (or wait for autovacuum to do it for you). <br />
<br />
Increasing the default_statistics_target may be useful but the default value shipped with PostgreSQL is a reasonable starting point.<br />
<br />
;PostgreSQL 8.3 and earlier<br />
<br />
In PostgreSQL 8.3 and earlier increasing the supplied default_statistics_target would often greatly improve query plans. The starting default_statistics_target value was raised from 10 to 100 in PostgreSQL 8.4. The maximum value for the parameter was also increased from 1000 to 10,000 in 8.4.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-WORK-MEM work_mem]==<br />
<br />
If you do a lot of complex sorts, and have a lot of memory, then increasing the <code>work_mem</code> parameter allows PostgreSQL to do larger in-memory sorts which, unsurprisingly, will be faster than disk-based equivalents.<br />
<br />
This size is applied to each and every sort done by each user, and complex queries can use multiple working memory sort buffers. Set it to 50MB, and have 30 users submitting queries, and you are soon using 1.5GB of real memory. Furthermore, if a query involves doing merge sorts of 8 tables, that requires 8 times work_mem. You need to consider what you set max_connections to in order to size this parameter correctly. This is a setting where data warehouse systems, where users are submitting very large queries, can readily make use of many gigabytes of memory.<br />
<br />
[http://www.postgresql.org/docs/9.3/static/runtime-config-logging.html#GUC-LOG-TEMP-FILES log_temp_files] can be used to log sorts, hashes, and temp files which can be useful in figuring out if sorts are spilling to disk instead of fitting in memory. You can see sorts spilling to disk using <code>EXPLAIN ANALYZE</code> plans as well. For example, if you see a line like <code>Sort Method: external merge Disk: 7526kB</code> in the output of EXPLAIN ANALYZE, a <code>work_mem</code> of at least 8MB would keep the intermediate data in memory and likely improve the query response time (although it may take substantially more than 8MB to do the sort entirely in memory, as data on disk is stored in a more compact format).<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM maintenance_work_mem]==<br />
<br />
Specifies the maximum amount of memory to be used by maintenance operations, such as VACUUM, CREATE INDEX, and ALTER TABLE ADD FOREIGN KEY. It defaults to 64 megabytes (64MB) since version 9.4. Since only one of these operations can be executed at a time by a database session, and an installation normally doesn't have many of them running concurrently, it's safe to set this value significantly larger than work_mem. Larger settings might improve performance for vacuuming and for restoring database dumps.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-wal.html#GUC-WAL-SYNC-METHOD wal_sync_method wal_buffers]==<br />
<br />
After every transaction, PostgreSQL forces a commit to disk out to its write-ahead log. This can be done a couple of ways, and on some platforms other options than the shipped wal_sync_method are considerably faster than the conservative default. open_sync is the most common non-default setting switched to, on platforms that support it but default to one of the fsync methods. See [http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm Tuning PostgreSQL WAL Synchronization] for a lot of background on this topic. Note that open_sync writing is buggy on some platforms (such as [http://lwn.net/Articles/350219/ Linux]), and you should (as always) do plenty of tests under a heavy write load to make sure that you haven't made your system less stable with this change. [[Reliable Writes]] contains more information on this topic.<br />
<br />
wal_buffers defaults to 1/32 of the size of shared_buffers, with an upper limit of 16MB (reached when shared_buffers=512MB). Adjustments to the default are required much less often than in earlier PostgreSQL releases.<br />
<br />
;PostgreSQL 9.0 and earlier<br />
<br />
Linux kernels starting with version 2.6.33 cause PostgreSQL versions before 9.0.2 to default to wal_sync_method=open_datasync; before kernel 2.6.32 the default picked was always fdatasync. This can cause a significant performance decrease when combined with small writes and/or small values for wal_buffers. PostgreSQL versions starting with 9.0.2 again default wal_sync_method to fdatasync when running on Linux.<br />
<br />
On PostgreSQL 9.0 and earlier, increasing wal_buffers from its tiny default of a small number of kilobytes is helpful for write-heavy systems. Benchmarking generally suggests that just increasing to 1MB is enough for some large systems, and given the amount of RAM in modern servers allocating a full WAL segment (16MB, the useful upper-limit here) is reasonable. <br />
<br />
Changing wal_buffers requires a database restart.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-query.html#GUC-CONSTRAINT-EXCLUSION constraint_exclusion]==<br />
<br />
<tt>constraint_exclusion</tt> now defaults to a new choice: <tt>partition</tt>. This will only enable constraint exclusion for partitioned tables which is the right thing to do in nearly all cases.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-MAX-PREPARED-TRANSACTIONS max_prepared_transactions]==<br />
<br />
This setting is used for managing 2 phase commit. If you do not use two phase commit (and if you don't know what it is, you don't use it), then you can set this value to 0. That will save a little bit of shared memory. For database systems with a large number (at least hundreds) of concurrent connections, be aware that this setting also affects the number of available lock-slots in pg_locks, so you may want to leave it at the default setting. There is a formula for how much memory gets allocated [http://www.postgresql.org/docs/current/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS in the docs] and in the default postgresql.conf.<br />
<br />
Changing max_prepared_transactions requires a server restart.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT synchronous_commit]==<br />
PostgreSQL can only safely use a write cache if it has a battery backup. See [http://www.postgresql.org/docs/current/static/wal-reliability.html WAL reliability] for an essential introduction to this topic. No, really; go read that right now, it's vital to understand that if you want your database to work right.<br />
<br />
You may be limited to approximately 100 transaction commits per second per client in situations where you don't have such a durable write cache (and perhaps only 500/second even with lots of clients).<br />
<br />
For situations where a small amount of data loss is acceptable in return for a large boost in how many updates you can do to the database per second, consider switching synchronous commit off. This is particularly useful in the situation where you do not have a battery-backed write cache on your disk controller, because you could potentially get thousands of commits per second instead of just a few hundred.<br />
<br />
For obsolete versions of PostgreSQL, you may find people recommending that you set ''fsync=off'' to speed up writes on busy systems. This is dangerous--a power loss could result in your database getting corrupted and not able to start again. Synchronous commit doesn't introduce the risk of ''corruption'', which is really bad, just some risk of data ''loss''.<br />
<br />
==[http://www.postgresql.org/docs/current/static/runtime-config-query.html#GUC-RANDOM-PAGE-COST random_page_cost]==<br />
This setting suggests to the optimizer how long it will take your disks to seek to a random disk page, as a multiple of how long a sequential read (with a cost of 1.0) takes. If you have particularly fast disks, as commonly found with RAID arrays of SCSI disks, it may be appropriate to lower random_page_cost, which will encourage the query optimizer to use random access index scans. Some feel that 4.0 is always too large on current hardware; it's not unusual for administrators to standardize on always setting this between 2.0 and 3.0 instead. In some cases that behavior is a holdover from earlier PostgreSQL versions where having random_page_cost too high was more likely to screw up plan optimization than it is now (and setting at or below 2.0 was regularly necessary). Since these cost estimates are just that--estimates--it shouldn't hurt to try lower values.<br />
<br />
But this not where you should start to search for plan problems. Note that random_page_cost is pretty far down this list (at the end in fact). If you are getting bad plans, this shouldn't be the first thing you look at, even though lowering this value may be effective. Instead, you should start by making sure autovacuum is working properly, that you are collecting enough statistics, and that you have correctly sized the memory parameters for your server--all the things gone over above. After you've done all those much more important things, if you're still getting bad plans ''then'' you should see if lowering random_page_cost is still useful.<br />
<br />
[[Category:Administration]] [[Category:Performance]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=PgBouncer&diff=34609PgBouncer2020-01-28T02:51:55Z<p>Barwick: Update PgBouncer website URL</p>
<hr />
<div>PgBouncer is a lightweight connection pooler for PostgreSQL.<br />
<br />
The information that used to be on this wiki page is contained on the [https://www.pgbouncer.org/ PgBouncer website] (https://www.pgbouncer.org/).</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=34174Repmgr2019-10-16T02:23:26Z<p>Barwick: Clarify supported PostgreSQL versions.</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL failover, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr helps DBAs and System Administrators manage a cluster of PostgreSQL databases. By taking advantage of the Hot Standby capability introduced in PostgreSQL 9, repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements.<br />
<br />
repmgr simplifies administration and daily management, enhances productivity and reduces the overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process<br />
* providing support for HA operations such as switch-overs and fail-overs.<br />
<br />
repmgr is available via 2ndQuadrant’s own package repositories as well as the PGDG community repositories.. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 5 is compatible with PostgreSQL 9.3 ~ PostgreSQL 12. See the [https://repmgr.org/docs/current/install-requirements.html#INSTALL-COMPATIBILITY-MATRIX repmgr compatibility matrix] for full details.<br />
<br />
<br />
== repmgr 5 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/current/release-5.0.html 5.0] (2019-10-15).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or from Barman<br />
* Timeline following<br />
** a standby can be promoted to a primary without requiring a restart<br />
** other standbys can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the primary to another standby mode<br />
* Replication slot support (PostgreSQL 9.4 and later), simplifying WAL retention management<br />
* [https://repmgr.org/docs/current/performing-switchover.html Switchover] support<br />
** supports role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at: https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Installation instructions for RedHat/Debian-based distributions are available [https://repmgr.org/docs/current/installation-packages.html here].<br />
<br />
== Documentation ==<br />
<br />
* [https://repmgr.org/docs/current/index.html repmgr documentation]<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=CommitFest&diff=33593CommitFest2019-06-28T06:13:38Z<p>Barwick: Update CF links from http to https</p>
<hr />
<div>A CommitFest (CF) is a periodic break to PostgreSQL development that focuses on patch review and commit rather than new development. These are held so that all the work for a release gets relatively prompt review and feedback, and so that work for a release doesn't pile up until the end of the release cycle. During these periods developers are asked to review and test the patches submitted by others. The hope is that this will also reduce the burden on those who do the final review and commit. The CF page is used to keep track of submitted patches and their status, to help manage the<br />
process.<br />
<br />
When we're not trying to put together a major release CFs tend to run for one month each with a one month gap between them. Due to the process of getting a release out the door, though, there may be several months in a row without such a review cycle. After such a hiatus the first CF may be immediately preceded by a review-only phase before committers are able to participate at normal levels; such a phase may be called a ReviewFest (RF).<br />
<br />
A [[Running_a_CommitFest|CommitFest manager]] arranges for all the [[Submitting a Patch|submitted patches]] to be [[Reviewing a Patch|reviewed]]. The peer review process often results in discussion on the pgsql-hackers list and/or requests for some sort of modification before commit. Most patches wind up getting committed, although some are returned with feedback (in hopes that the submitter will make some change and submit to a later cycle) or rejected (if they are determined by the community not to be useful changes). Once complete, a new [[Alpha release process|Alpha release]] is produced for testing; after the final CommitFest for a release, a beta version is packaged.<br />
<br />
The CommitFests are managed through our web application at [https://commitfest.postgresql.org commitfest application]. Please submit/review/organise patches there.<br />
<br />
* If you want to submit a new patch for review, please visit [https://commitfest.postgresql.org/action/commitfest_view/open CommitFest Open].<br />
* If you want to help with the reviewing process for the commitfest that's currently taking place, please visit [https://commitfest.postgresql.org/action/commitfest_view/inprogress CommitFest In-Progress]<br />
* If you want to view the most recently closed commitfest, please visit [https://commitfest.postgresql.org/action/commitfest_view/previous CommitFest Previous]<br />
<br />
[[Category:CommitFest]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=CommitFest&diff=33592CommitFest2019-06-28T06:13:10Z<p>Barwick: Update CF links from http to https</p>
<hr />
<div>A CommitFest (CF) is a periodic break to PostgreSQL development that focuses on patch review and commit rather than new development. These are held so that all the work for a release gets relatively prompt review and feedback, and so that work for a release doesn't pile up until the end of the release cycle. During these periods developers are asked to review and test the patches submitted by others. The hope is that this will also reduce the burden on those who do the final review and commit. The CF page is used to keep track of submitted patches and their status, to help manage the<br />
process.<br />
<br />
When we're not trying to put together a major release CFs tend to run for one month each with a one month gap between them. Due to the process of getting a release out the door, though, there may be several months in a row without such a review cycle. After such a hiatus the first CF may be immediately preceded by a review-only phase before committers are able to participate at normal levels; such a phase may be called a ReviewFest (RF).<br />
<br />
A [[Running_a_CommitFest|CommitFest manager]] arranges for all the [[Submitting a Patch|submitted patches]] to be [[Reviewing a Patch|reviewed]]. The peer review process often results in discussion on the pgsql-hackers list and/or requests for some sort of modification before commit. Most patches wind up getting committed, although some are returned with feedback (in hopes that the submitter will make some change and submit to a later cycle) or rejected (if they are determined by the community not to be useful changes). Once complete, a new [[Alpha release process|Alpha release]] is produced for testing; after the final CommitFest for a release, a beta version is packaged.<br />
<br />
The CommitFests are managed through our web application at [http://commitfest.postgresql.org commitfest application]. Please submit/review/organise patches there.<br />
<br />
* If you want to submit a new patch for review, please visit [https://commitfest.postgresql.org/action/commitfest_view/open CommitFest Open].<br />
* If you want to help with the reviewing process for the commitfest that's currently taking place, please visit [https://commitfest.postgresql.org/action/commitfest_view/inprogress CommitFest In-Progress]<br />
* If you want to view the most recently closed commitfest, please visit [https://commitfest.postgresql.org/action/commitfest_view/previous CommitFest Previous]<br />
<br />
[[Category:CommitFest]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=PostgreSQL_12_Open_Items&diff=33573PostgreSQL 12 Open Items2019-06-17T14:32:54Z<p>Barwick: /* Open Issues */ postgresql.auto.conf issue</p>
<hr />
<div>== Open Issues ==<br />
<br />
'''NOTE''': Please add new open items to the bottom of the list.<br />
<br />
* [https://www.postgresql.org/message-id/21516.1552489217@sss.pgh.pa.us Debate INFO messages in ATTACH PARTITION and SET NOT NULL]<br />
* [https://www.postgresql.org/message-id/CAD21AoCqs8iN04RX=i1KtLSaX5RrTEM04b7NHYps4+rqtpWNEg@mail.gmail.com Add vacuum_index_cleanup for toast relations?]<br />
** Commit: {{PgCommitURL|a96c41feec6b6616eb9d5baee9a9e08c20533c38}}, Author: Masahiko Sawada, Owner: Robert Haas<br />
* [https://www.postgresql.org/message-id/10797.1552679128@sss.pgh.pa.us Memory leak associated with dubious RelationData.rd_pdcxt handling]<br />
** Commit: {{PgCommitURL|898e5e3290a72d288923260143930fb32036c00c}}, Author: Robert Haas, Owner: Robert Haas<br />
* [https://www.postgresql.org/message-id/68d00017-ae5c-b14f-fc3a-c9e38e3ce792%40iki.fi GiST Page can become unrecyclable due to PageDeleteXid wraparound]<br />
** Commit {{PgCommitURL|7df159a620b760e289f1795b13542ed1b3e13b87}} Author: Heikki Linnakangas<br />
* [https://www.postgresql.org/message-id/20190527203713.GA58392@gust.leadboat.com \connect uses the same IP as the existing connection, docs no longer match behavior, etc.]<br />
** Commit {{PgCommitURL|6e5f8d4}} Author: Fabien Coelho<br />
* [https://www.postgresql.org/message-id/20190607165105.vn4bl6piofroj3um@alap3.anarazel.de BulkInsertStates and copy.c with partitioned tables]<br />
* [https://www.postgresql.org/message-id/20190611061115.njjwkagvxp4qujhp%40alap3.anarazel.de check_recovery_target_lsn() does a PG_CATCH without a throw]<br />
** Commit {{PgCommitURL|2dedf4d9a899b36d1a8ed29be5efbd1b31a8fe85}} Author: Peter Eisentraut<br />
* [https://postgr.es/m/CAPpHfdvGVegF_TKKRiBrSmatJL2dR9uwFCuR%2BteQ_8tEXU8mxg%40mail.gmail.com Hash join explain can fail with "bogus varno: 65000"]<br />
** Commit {{PgCommitURL|5f32b29c18195299e90c1fb6c8945e9a46d772d2}} Author: Andres Freund<br />
* [https://www.postgresql.org/message-id/aed6cc9f-98f3-2693-ac81-52bb0052307e%402ndquadrant.com postgresql.auto.conf file with duplicate entries not handled properly]<br />
<br />
== Decisions to Recheck Mid-Beta ==<br />
<br />
== Older Bugs ==<br />
<br />
=== Live issues ===<br />
<br />
* [https://www.postgresql.org/message-id/15672-b9fa7db32698269f%40postgresql.org ATPostAlterTypeCleanup causes child indexes to be recreated with wrong relfilenode]<br />
** Crash/data corruption is fixed by {{PgCommitURL|02c359eeda50a71c951371c9d3e920ff8f514008}}<br />
** There's more to be done here, but it's not clear whether additional work is small enough to be in-scope for v11 or v12<br />
* [https://www.postgresql.org/message-id/15726-6d67e4fa14f027b3@postgresql.org parallel queries failed ERROR: invalid name syntax CONTEXT: parallel worker]<br />
* [https://www.postgresql.org/message-id/15746-6e0482a4c0f915cb@postgresql.org BUG #15746: cache lookup failed for function in plpgsql block]<br />
** This was already fixed in HEAD by a part of {{PgCommitURL|04fe805a1734eccd8dcdd34c8cc0ddcb62c7240c}}<br />
** Issue is whether it's worth the risk to back-patch unproven code<br />
* [https://www.postgresql.org/message-id/CA+hUKGKVWbz_iniqvFujPZLioFPxGwuVV6PJeeCrQ8SVcdg7FQ@mail.gmail.com Change resowner cleanup order for Windows?]<br />
<br />
=== Fixed issues ===<br />
<br />
* [https://www.postgresql.org/message-id/20181009.181536.142257785.horiguchi.kyotaro@lab.ntt.co.jp Bypass processing of wraparound autovacuums not marked as aggressive]<br />
** Problem exists since the point where aggressive vacuums have been introduced, v12 has only added extra logs to look after the impossible case of wraparound autovacuums not aggressive.<br />
** Fixed in: {{PgCommitURL|2aa6e331ead7f3ad080561495ad4bd3bc7cd8913}}<br />
* [https://www.postgresql.org/message-id/15733-7692379e310b80ec%40postgresql.org An insert destined at partition created after a column has been dropped from the parent table fails]<br />
** Fixed in: {{PgCommitURL|6b0208ebc436b33bd80ce264299b4b1b8d59b68a}}<br />
* [https://www.postgresql.org/message-id/7961.1552498252%40sss.pgh.pa.us RelationData.rd_partcheck should get its own memory context]<br />
** Fixed in: {{PgCommitURL|5f1433ac5e7f943b29ef01266b6b8fc915e6b917}}<br />
* [https://www.postgresql.org/message-id/15734-2daa8761eeed8e20@postgresql.org Walsender process crashing when executing SHOW ALL]<br />
** Fixed in: {{PgCommitURL|c34677fdaa73f089d557554a9cd479b9bd5b5143}}<br />
* [https://www.postgresql.org/message-id/016deb6b-1f0a-8e9f-1833-a8675b170aa9@postgresql.org Possible to store invalid SCRAM-SHA-256 Passwords]<br />
** Fixed in: {{PgCommitURL|ccae190b916f27fbe4079ee4664d34cd1be47b79}}<br />
* [https://www.postgresql.org/message-id/15781-2601b1002bad087c@postgresql.org BUG #15781: subselect on foreign table (postgres_fdw) can crash]<br />
** Fixed in: {{PgCommitURL|8cad5adb9c0be82e9f40d51b02a542439f47de9e}}<br />
* [https://www.postgresql.org/message-id/9813f079-f16b-61c8-9ab7-4363cab28d80@lab.ntt.co.jp selecting from partition directly can't use constraint exclusion]<br />
** Fixed in: {{PgCommitURL|e03ff739695cb731956763355e8e0f38c6905008}}<br />
* [https://www.postgresql.org/message-id/20190418011430.GA19133@paquier.xyz REINDEX INDEX on an index of pg_class can fail]<br />
** Fixed in: {{PgCommitURL|f912d7dec29341d55315fccef8dc3fdfd068c6e3}}<br />
* [https://www.postgresql.org/message-id/FAD28A83-AC73-489E-A058-2681FA31D648@tvsquared.com Partition pruning is broken for stable WHERE conditions]<br />
** Fixed in: {{PgCommitURL|6630ccad7a25cad32e2d1a6833fb971602cb67fe}} and predecessor commits<br />
* [https://www.postgresql.org/message-id/flat/CAKcux6%3DuZEyWyLw0N7HtR9OBc-sWEFeByEZC7t-KDf15FKxVew%40mail.gmail.com Statistical aggregate functions are not working with partitionwise aggregate]<br />
** Fixed in: {{PgCommitURL|2657283256f1cab53d09d2c7db1ce9b7065193a0}}<br />
* [https://www.postgresql.org/message-id/20190416070119.GK2673@paquier.xyz Race conditions with checkpointer and shutdown]<br />
** Fixed in {{PgCommitURL|a1a789eb5ac894b4ca4b7742f2dc2d9602116e46}}<br />
** Back-patched to v10; back-patching further is unattractive from both risk and work-required standpoints<br />
<br />
=== Nothing to do ===<br />
<br />
* [https://www.postgresql.org/message-id/20190403063759.GF3298@paquier.xyz toast_tuple_target reloption doesn't work as expected]<br />
** The consensus would be to increase the upper boundary of toast_tuple_target, but this means potentially breaking a category of dumps.<br />
<br />
== Non-bugs ==<br />
<br />
* [https://www.postgresql.org/message-id/CAD21AoB_+PSoO4J2dKEgy9qKf2uNnbHHOOSUcz6f20f-=T-bdg@mail.gmail.com vacuumdb and new VACUUM options]<br />
== Resolved Issues ==<br />
<br />
=== resolved before 12beta2 ===<br />
<br />
* [https://www.postgresql.org/message-id/CAGPqQf0cYjm1%3Drjxk_6gU0SjUS70%3DyFUAdCJLwWzh9bhNJnyVg%40mail.gmail.com CREATE TABLE .. PARTITION OF doesn't respect default_tablespace]<br />
** Fixed in: {{PgCommitURL|a36c84c3e4a9bee6baa7}}<br />
* [https://www.postgresql.org/message-id/CALAY4q99FcFCoG6ddke0V-AksGe82L_+bhDWgEfgZBakB840zA@mail.gmail.com with oids option not removed in pg_dumpall]<br />
** Commit {{PgCommitURL|578b229718e8f15fa779e20f086c4b6bb3776106}}<br />
** Fixed in: {{PgCommitURL|657c2384c6c79c6ed0d6f71f811b2fc7c41f104a}}<br />
* [https://www.postgresql.org/message-id/20190522083038.GA16837@paquier.xyz pg_dump throwing "column number -1 is out of range 0..36" on HEAD]<br />
** Fixed in: {{PgCommitURL|54487d1560619a0027e0651d1b8d715ca8fc388c}}<br />
* [https://www.postgresql.org/message-id/CA%2BrenyUuSmYgmZjKc_DfUNVZ0uttF91-FwhDVW3F7WEPj0jL5w%40mail.gmail.com ddl.sgml still says foreign keys can't point to partitioned tables]<br />
** Commit {{PgCommitURL|f56f8f8da6afd8523b4d5284e02a20ed2b33ef8d}} Author: Alvaro Herrera<br />
** Fixed in: {{PgCommitURL|f73293aba4d43e48707e361b2b1ef1465fef46e0}}<br />
* [https://www.postgresql.org/message-id/20190601191007.GC1905@paquier.xyz psql completion bugs with access methods]<br />
** Fixed in: {{PgCommitURL|0240a00fbd4fd14f577edf8d36a032237fd0b9cb}}<br />
* [https://www.postgresql.org/message-id/15832-b1bf336a4ee246b5@postgresql.org COPY into a partitioned table breaks its indexes]<br />
** Fixed in: {{PgCommitURL|56b3b3838284f53c83556592e60688522155f57f}}<br />
* [https://www.postgresql.org/message-id/20190607043415.GE1736@paquier.xyz be-gssapi-common.h not in correct location]<br />
** Fixed in: {{PgCommitURL|35b2d4bc0eb5d61a2a294ccb6b2e4abdad307604}}<br />
* [https://www.postgresql.org/message-id/CAJrrPGcAxsMM7n__HJRPBrh7Y6ruU6LetfPD=cPGeW=G49na0g@mail.gmail.com pg_basebackup failure after setting default_table_access_method option]<br />
** Fixed in: {{PgCommitURL|fff2a7d7bd09db38e1bafc1303c29b10a9805dc0}}<br />
* [https://www.postgresql.org/message-id/CALfoeiugyrXZfX7n0ORCa4L-m834dzmaE8eFdbNR6PMpetU4Ww%40mail.gmail.com Inconsistency between table am callback and table function names]<br />
** many commits, Author: Andres Freund<br />
** Fixed in: {{PgCommitURL|73b8c3bd2889fed986044e15aefd0911f96ccdd3}}<br />
* [https://www.postgresql.org/message-id/CAKJS1f-2rx+E9mG3xrCVHupefMjAp1+tpczQa9SEOZWyU7fjEA@mail.gmail.com Documents don't warn about using too many partitions]<br />
** Fixed in: {{PgCommitURL|e788e849addd56007a0e75f3b5514f294a0f3bca}}<br />
* [https://www.postgresql.org/message-id/CAEZATCUhT9rt7Ui%3DVdx4N%3D%3DVV5XOK5dsXfnGgVOz_JhAicB%3DZA%40mail.gmail.com Multivariate MCV stats can leak data to unprivileged users]<br />
** Fixed by {{PgCommitURL|6cbfb784c3c91146148a76d50cda6f69ae6a79fb}} et seq<br />
<br />
=== resolved before 12beta1 ===<br />
<br />
* [https://www.postgresql.org/message-id/20190225074539.az6j3u464cvsoxh6@depesz.com Segfault when restoring -Fd dump on current HEAD]<br />
** Fixed in {{PgCommitURL|7fcdb5e0021}}<br />
* [https://www.postgresql.org/message-id/CAKJS1f_1c260nOt_vBJ067AZ3JXptXVRohDVMLEBmudX1YEx-A@mail.gmail.com pg_dump is broken for partition tablespaces]<br />
** Fixed in commits: {{PgCommitURL|87259588d0ab0b8e742e30596afa7ae25caadb18}}<br />
** and {{PgCommitURL|3b23552ad8bb}}<br />
* [https://www.postgresql.org/message-id/15727-0be246e7d852d229@postgresql.org PANIC: cannot abort transaction XXX, it was already committed]<br />
** One issue fixed in: {{PgCommitURL|41f5e04aec6cf63ba8392adf70e9289e9c3706d6}}<br />
** Another issue fixed in: {{PgCommitURL|f7feb020c3d8d5aff24204af28359b99ee65bf8f}}<br />
* [https://www.postgresql.org/message-id/201902021315.6h6ktmmsgjmx@alvherre.pgsql remove \cset from pgbench]<br />
** Fixed in: {{PgCommitURL|25ee70511ec2ccbef0ad3fe64875a4d552cdcd50}}<br />
* [https://www.postgresql.org/message-id/20190322032612.GA323@alvherre.pgsql pg_partition_root crashes when using top-most parent in input]<br />
** Fixed in: {{PgCommitURL|2ab6d28d233af17987ea323e3235b2bda89b4f2e}}<br />
* [https://www.postgresql.org/message-id/CA+HiwqEGoa485g18mt9GUdF8fH4mKDgpeoc32XiW-dRUFpN5Lw@mail.gmail.com Server crash in transformPartitionRangeBounds]<br />
** Fixed in: {{PgCommitURL|cdde886d36b5a4d7ad9e1d02596f7fa1c8c129e3}}<br />
* [https://www.postgresql.org/message-id/20190326020853.GM2558@paquier.xyz Misleading errors with column references in default expressions and partition bounds]<br />
** Fixed in: {{PgCommitURL|ecfed4a12247cf4659eee6b6ea27405e35fe57f8}}<br />
* [https://www.postgresql.org/message-id/8305.1553884377@sss.pgh.pa.us Planner's partitionwise-join code crashes under GEQO]<br />
** Fixed in: {{PgCommitURL|7ad6498fd5a654de6e743814c36cf619a3b5ddb6}}<br />
* [https://www.postgresql.org/message-id/flat/19465.1541636036@sss.pgh.pa.us Inadequate index locking causes Assert failure]<br />
** Fixed in: {{PgCommitURL|9c703c169a872d144f2f79d2fb211c82587adfa7}}<br />
* [https://www.postgresql.org/message-id/87wolmg60q.fsf@news-spur.riddles.org.uk Inlining of nested CTEs with recursive terms]<br />
** Fixed in: {{PgCommitURL|9476131278c7bfc435ad9a21fc8e981272ac0dd2}}<br />
* [https://www.postgresql.org/message-id/DF4PR8401MB11964EDB77C860078C343BEBEE5A0@DF4PR8401MB1196.NAMPRD84.PROD.OUTLOOK.COM Indexes part of a partition tree cannot be run with REINDEX CONCURRENTLY]<br />
** Fixed in: {{PgCommitURL|ef6f30fe77af69a8c775cca82bf993b10c9889ee}}<br />
* [https://www.postgresql.org/message-id/flat/CABUevEzD_duH_hGyZw14o%2BkhHBw-rWSSAxbEKt5HWy2cK0Djdw%40mail.gmail.com#d8a9d175134a072dd1477c3fac96f76a Keep track of checksum failures in shared object, last failure time and pg_stat_checkums view]<br />
** Commit: {{PgCommitURL|6b9e875f7286d8535bff7955e5aa3602e188e436}}, Author: Magnus Hagander, Owner: Magnus Hagander<br />
** Fixed in: {{PgCommitURL|77bd49adba4711b4497e7e39a5ec3a9812cbd52a}}<br />
* [https://www.postgresql.org/message-id/5CAF3B8F.2090905@lab.ntt.co.jp Partition tuple routing code does not always call EndForeignInsert]<br />
** Commit: {{PgCommitURL|3f2393edefa5ef2b6970a5a2fa2c7e9c55cc10cf}}, Author: David Rowley, Amit Langote and Álvaro Herrera: Owner: Álvaro Herrera<br />
** Fixed in: {{PgCommitURL|3a45321a491711b556d2cf8f6904ab989b9d0b08}}<br />
* [https://www.postgresql.org/message-id/a620f85a-42ab-e0f3-3337-b04b97e2e2f5%40redhat.com COLLATE: Hash partition vs UPDATE]<br />
** Fixed in: {{PgCommitURL|4b40e44f07c727c7a82b291d3b60098dd99f3f64}}<br />
* [https://www.postgresql.org/message-id/20190411134947.GA22043@alvherre.pgsql Consider invalid indexes for REINDEX INDEX CONCURRENTLY?]<br />
** Fixed in: {{PgCommitURL|a6dcf9df4d91ff0db23579f9114079abe6f3e2bf}}<br />
* [https://www.postgresql.org/message-id/366.1555382816@sss.pgh.pa.us ExecForceStoreMinimalTuple leaks memory like there's no tomorrow]<br />
** Commit: {{PgCommitURL|4da597edf1bae0cf0453b5ed6fc4347b6334dfe1}}, Author: Andres Freund, Ashutosh Bapat, Owner: Andres Freund<br />
** Fixed in {{PgCommitURL|88e6ad3054ddd5aa0dee12e5def2c335fe92a414}}<br />
* [https://www.postgresql.org/message-id/CAOuzzgqS-CL18_zKF7pF-wymG8mUeUZveNYYSrXKQRn1VaJsug@mail.gmail.com GSSAPI encryption missing protocol documentation]<br />
** Commit: {{PgCommitURL|b0b39f72b9904bcb80f97b35837ccff1578aa4b8}}, Author: Robbie Harwood, Owner: Stephen Frost<br />
** Fixed in {{PgCommitURL|eb882a1b717589777e068dc6671830226f3aa7f0}}<br />
* [https://www.postgresql.org/message-id/8416d227-6e9d-092a-4475-b453e1d7d433@2ndquadrant.com New regression tests for GSSAPI encryption are unstable]<br />
** Commit: {{PgCommitURL|b0b39f72b9904bcb80f97b35837ccff1578aa4b8}}, Author: Robbie Harwood, Owner: Stephen Frost<br />
** Fixed in {{PgCommitURL|eb882a1b717589777e068dc6671830226f3aa7f0}}<br />
* [https://www.postgresql.org/message-id/flat/20190330224333.GQ5815%40telsasoft.com clean up docs for v12]<br />
** Fixed in {{PgCommitURL|148266fa354a47543f6c0325cd1ea900ead4aac6}}<br />
* [https://www.postgresql.org/message-id/CAH2-Wzm08nr+JPx4jMOa9CGqxWYDQ-_D4wtPBiKghXAUiUy-nQ@mail.gmail.com Pathological performance when inserting NULL values into unique index]<br />
** Commit: {{PgCommitURL|dd299df8189bd00fbe54b72c64f43b6af2ffeccd}}, Author: Peter Geoghegan, Owner: Peter Geoghegan<br />
** Fixed in {{PgCommitURL|9b10926263d831fac5758f1493c929a49b55669b}}<br />
* [https://www.postgresql.org/message-id/20190408002847.GA904@telsasoft.com Cleanup/remove/update references to OID column]<br />
** Commit: {{PgCommitURL|578b229718e8f15fa779e20f086c4b6bb3776106}}, Author: Andres Freund, Owner: Andres Freund<br />
** Fixed in {{PgCommitURL|f6b39171f3d65155b9390c2c69bc5b3469f923a8}}<br />
* [https://www.postgresql.org/message-id/15751.1555256860@sss.pgh.pa.us topminnow triggered assertion failure with vacuum_index_cleanup]<br />
** Fixed in {{PgCommitURL|dd69597988859c51131e0cbff3e30432db4259e1}}<br />
* [https://www.postgresql.org/message-id/CAMkU=1x8taZfsbPkv_MsWbTtzibW_yQHXoMhF_DTtm=z2hVHDg@mail.gmail.com compiler warning in pgcrypto imath.c]<br />
** Commit: {{PgCommitURL|48e24ba6b7fd3bfd156b51e8d768fd48df0d288b}}, Author: Noah Misch, Owner: Noah Misch<br />
** Fixed in {{PgCommitURL|726cc4242a2f766c8280a72ef7c8418965d139c8}}<br />
* [https://www.postgresql.org/message-id/CAHGQGwHa_dX%3DdRcbR5QVTs6W5mgCy3qZ2fEwREaiXpES1B2%2Bjw%40mail.gmail.com Add TRUNCATE option to vacuum command as well as reloption]<br />
** Commit: {{PgCommitURL|119dcfad988d5b5d9f52b256087869997670aa36}}, Author: Tsunakawa Takayuki, Owner: Fujii Masao<br />
** Fixed in {{PgCommitURL|b84dbc8eb80b43e554891c459a19969d9fbdefe5}}<br />
* [https://www.postgresql.org/message-id/20190416180452.3pm6uegx54iitbt5@alap3.anarazel.de Improvements in no-fsm-for-small-rels patch suggested by Andres Freund]<br />
** Commit: {{PgCommitURL|b0eaa4c51bbff3e3c600b11e5d104d6feb9ca77f}}, Author: John Naylor, Amit Kapila, Owner: Amit Kapila<br />
** Fixed in {{PgCommitURL|7db0cde6b58eef2ba0c70437324cbc7622230320}}<br />
* [https://www.postgresql.org/message-id/16170.1557251214@sss.pgh.pa.us Leakage of predicate locks]<br />
** Fixed in {{PgCommitURL|47a338cfcd67139a1f91892b080934fcfc3aea03}}<br />
* [https://www.postgresql.org/message-id/20190430151735.wi52sxjvxsjvaxxt@alap3.anarazel.de Heap lock levels for REINDEX INDEX CONCURRENTLY not quite right?]<br />
** Commit: {{PgCommitURL|5dc92b844e680c54a7ecd68de0ba53c949c3d605}}, Author: Michael Paquier, Owner: Peter Eisentraut<br />
** Fixed in {{PgCommitURL|add85ead4ab969c1e31d64f4476c2335856f4aa9}}<br />
* [https://www.postgresql.org/message-id/23694.1556806002@sss.pgh.pa.us Inconsistent error message wording for REINDEX CONCURRENTLY]<br />
** Commit: {{PgCommitURL|5dc92b844e680c54a7ecd68de0ba53c949c3d605}}, Author: Michael Paquier, Owner: Peter Eisentraut<br />
** Fixed in {{PgCommitURL|508300e2e141a9fd87758ce01374c5b0597986fd}}<br />
* [https://www.postgresql.org/message-id/CA+hUKGJRzLo7tZExWfSbwM3XuK7aAK7FhdBV0FLkbUG+W0v0zg@mail.gmail.com Wrong answers from queries using a GIST index]<br />
** Commit: {{PgCommitURL|9155580fd5fc2a0cbb23376dfca7cd21f59c2c7b}}, Author: Anastasia Lubennikova, Andrey V. Lepikhov, Owner: Heikki Linnakangas<br />
** Fixed in {{PgCommitURL|22251686f07f70527aecda22ab5402986884f6f5}}<br />
* [https://www.postgresql.org/message-id/a912ffff-f6e4-778a-c86a-cf5c47a12933@2ndquadrant.com Circular dependency between libpgcommon and libpgfeutils]<br />
** Fixed in {{PgCommitURL|fc9a62af3f87f4bec1e8c904ea99ae50f3c881ef}}<br />
* [https://www.postgresql.org/message-id/CA%2BTgmoZP-CTmEPZdmqEOb%2B6t_Tts2nuF7eoqxxvXEHaUoBDmsw%40mail.gmail.com Should effective_io_concurrency + 10 be used for an index's page deletion table scans, or a new GUC]<br />
** Decision: leave the "+ 10" in for v12, and revisit in a later release when we have a better idea what kind of GUCs we want to control prefetching here and elsewhere<br />
* [https://www.postgresql.org/message-id/CAE9k0P=FvcDswnSVtRpSyZMpcAWC=Gp=ifZ0HdfPaRQ=__LBtw@mail.gmail.com Passing CopyMultiInsertInfo structure to CopyMultiInsertInfoNextFreeSlot()]<br />
** Commit {{PgCommitURL|86b85044e823a304d2a265abc030254d39efe7df}} Author: David Rowley, Andres Freund<br />
** Decision: it's ok to have the unused parameter<br />
* [https://www.postgresql.org/message-id/15804-3721117bf40fb654@postgresql.org Assertion failure when using logging_collector on Windows]<br />
** Commit {{PgCommitURL|57431a911d3a650451d198846ad3194900666152}} Author: Peter Eisentraut<br />
** Fixed for 12beta1 by reverting, in {{PgCommitURL|833451552925d0175e1e15128e411ddef9a36996}}, the necessary changes are too big for v12<br />
<br />
== Important Dates ==<br />
<br />
Current schedule:<br />
* feature freeze: April 7, 2019<br />
* beta1: May 23, 2019<br />
* beta2: June 20, 2019<br />
* beta3: XXX<br />
* rc1: XXX<br />
* ga: XXX<br />
<br />
[[Category:Open_Items]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=User:Barwick&diff=33560User:Barwick2019-06-10T08:14:20Z<p>Barwick: link fixes</p>
<hr />
<div>Ian Barwick. DBA and developer based in Tokyo, Japan. PostgreSQL user since 2001 (version 7.1). [https://sql-info.de/postgresql/notes/index.html My PostgreSQL blog]<br />
<br />
==Bookmarks==<br />
<br />
* [[BDR Project]]<br />
** [[BDR User Guide]]<br />
* [[Event Triggers]]<br />
** [[Audit trigger 91plus]]<br />
* [[Foreign data wrappers]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=33559Foreign data wrappers2019-06-10T08:08:49Z<p>Barwick: /* Specific SQL Database Wrappers */ update firebird_fdw</p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained ?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|}<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/postgres-fdw.html documentation]<br />
|<br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
|<br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|FDW for MySQL<br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://www.firebirdsql.org/ Firebird]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|[https://github.com/ibarwick/firebird_fdw/blob/master/README.md README]<br />
|version 1.1 released (2019-05)<br />
|-<br />
|SQLite<br />
|Native<br />
|<br />
|[https://github.com/gleu/sqlite_fdw github]<br />
|<br />
|<br />
|An FDW for SQLite3 (read-only)<br />
|-<br />
|[https://www.sqlite.org/index.html SQLite]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pgspider/sqlite_fdw github]<br />
|[https://pgxn.org/dist/sqlite_fdw PGXN]<br />
|[https://github.com/pgspider/sqlite_fdw/blob/master/README.md README]<br />
|An FDW for SQLite3 (write support and several pushdown optimization)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[https://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[https://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
| [https://github.com/griddb/griddb_nosql GridDB]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/griddb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/griddb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| InfluxDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/influxdb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/influxdb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|[https://multicorn.org/ Multicorn]<br />
|GPLv3<br />
|[https://github.com/sim51/neo4j-fdw Github]<br />
|<br />
|[https://github.com/sim51/neo4j-fdw/blob/master/README.adoc README]<br />
|FWD for Neo4j and also add a Cypher function to Pg<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/slamdata/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to]<br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| CSV / gzipped<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
|<br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
|<br />
|-<br />
| Multi-File<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| XML<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [https://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [https://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [https://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
|<br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
|<br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
|<br />
|}<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ICAL<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
|<br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
|<br />
|-<br />
| IMAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
|<br />
|-<br />
| RSS<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed.<br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Database.com<br />
| [https://multicorn.org/ Multicorn]<br />
| BSD<br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Dun & Badstreet<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
|<br />
|<br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Facebook<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
|<br />
| [https://github.com/hakanensari/frankfurter GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Keycloak<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/schne324/foreign-keycloak-wrapper GitHub]<br />
| [https://pgxn.org/dist/foreign-keycloak-wrapper/ PGXN]<br />
| [https://github.com/schne324/foreign-keycloak-wrapper/blob/master/README.md README]<br />
| Direct database integration with the [https://www.keycloak.org Keycloak] open-source Identity/Access Management solution.<br />
|-<br />
| Mailchimp<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
|<br />
|<br />
| Beta<br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| S3CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
|<br />
|<br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| Telegram<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/telegram_fdw GitHub]<br />
|<br />
|<br />
| telegram_fdw is a Telegram BOT implemented using the PostgreSQL foreign data wrapper interface.<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
|<br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
|<br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [https://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elastic Search<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/Mikulas/pg-es-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google BigQuery<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
|<br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
|<br />
|<br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://www.openscg.com/bigsql/hadoopfdw/ Bitbucket]<br />
|<br />
|<br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Hive<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
|<br />
|<br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://impala.apache.org/ Impala]<br />
| Native<br />
| BSD<br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://arrow.apache.org/ Apache Arrow]<br />
| Native<br />
| GPLv2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
|<br />
| A part of PG-Strom feature; as a columnar data source with support of SSD-to-GPU Direct SQL <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
|<br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
|<br />
|<br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Log files<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OpenStack / Telemetry<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OS Query<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
|<br />
|<br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
|<br />
|<br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
|<br />
|<br />
| A foreign datawrapper for querying system stats based on [https://libstatgrab.org/ statgrab]<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| faker_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/faker_fdw GitHub]<br />
|<br />
|<br />
| faker_fdw is a foreign data wrapper for PostgreSQL that generates fake data.<br />
|-<br />
| fdw_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
|<br />
|<br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PPG<br />
| Native<br />
|<br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
|<br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www2.meethue.com/en-us/philips-hue-benefits Phillips Hue Lighting Systems]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Random Number<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
|<br />
|<br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| [https://multicorn.org/ Multicorn]<br />
| Native<br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
| PostgreSQL<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator<br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
|<br />
|<br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
|<br />
|<br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Black Hole<br />
|<br />
|<br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
|<br />
|<br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [https://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=33206Repmgr2019-04-03T02:08:01Z<p>Barwick: Add link to current documentation version</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL failover, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr helps DBAs and System Administrators manage a cluster of PostgreSQL databases. By taking advantage of the Hot Standby capability introduced in PostgreSQL 9, repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements.<br />
<br />
repmgr simplifies administration and daily management, enhances productivity and reduces the overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process<br />
* providing support for HA operations such as switch-overs and fail-overs.<br />
<br />
repmgr is available via 2ndQuadrant’s own package repositories as well as the PGDG community repositories.. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 4.x is compatible with all PostgreSQL versions from PostgreSQL 9.3 onwards.<br />
<br />
<br />
== repmgr 4 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/current/release-4.3.html 4.3] (2019-04-02).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or from Barman<br />
* Timeline following<br />
** a standby can be promoted to a primary without requiring a restart<br />
** other standbys can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the master to another standby mode<br />
* Replication slot support (PostgreSQL 9.4 and later), simplifying WAL retention management<br />
* [https://repmgr.org/docs/current/performing-switchover.html Switchover] support<br />
** supports role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at: https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Installation instructions for RedHat/Debian-based distributions are available [https://repmgr.org/docs/current/installation-packages.html here].<br />
<br />
== Documentation ==<br />
<br />
* [https://repmgr.org/docs/current/index.html repmgr documentation]<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=33205Repmgr2019-04-03T02:05:21Z<p>Barwick: /* repmgr 4 Features */ update links</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL failover, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr helps DBAs and System Administrators manage a cluster of PostgreSQL databases. By taking advantage of the Hot Standby capability introduced in PostgreSQL 9, repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements.<br />
<br />
repmgr simplifies administration and daily management, enhances productivity and reduces the overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process<br />
* providing support for HA operations such as switch-overs and fail-overs.<br />
<br />
repmgr is available via 2ndQuadrant’s own package repositories as well as the PGDG community repositories.. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 4.x is compatible with all PostgreSQL versions from PostgreSQL 9.3 onwards.<br />
<br />
<br />
== repmgr 4 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/current/release-4.3.html 4.3] (2019-04-02).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or from Barman<br />
* Timeline following<br />
** a standby can be promoted to a primary without requiring a restart<br />
** other standbys can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the master to another standby mode<br />
* Replication slot support (PostgreSQL 9.4 and later), simplifying WAL retention management<br />
* [https://repmgr.org/docs/current/performing-switchover.html Switchover] support<br />
** supports role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at: https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Installation instructions for RedHat/Debian-based distributions are available [https://repmgr.org/docs/4.2/installation-packages.html here].<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Pgbench&diff=33107Pgbench2019-03-05T01:06:04Z<p>Barwick: update some URLs</p>
<hr />
<div>==pgbench==<br />
<br />
[https://www.postgresql.org/docs/current/pgbench.html pgbench] is a popular benchmarking tool used by many developers and hackers to do quick performance test with PostgreSQL on a system setup.<br />
<br />
On this page are recommendations from community on best ways to test with pgbench on a relatively bigger scale so that the frame of reference is consistent for the rest of the community reading into the results of pgbench<br />
<br />
==pgbench options==<br />
* -s = At minimum should be greater than the number of concurrent clients (-c) option and typically be bigger so as to be more meaningful<br />
* -c = Generally pgbench client becomes cpu limited somewhere between 150-250 concurrent users. Also using less than 2 will not generally show contentions in lock much<br />
* -t = Should be reasonably high. For example "HOT" benefit of limiting bloat in 8.3 is not clearly visible if the benchmark ran for short duration of time<br />
<br />
==Additional helpful information about parameters==<br />
* [http://archive.is/5UwS6 Determining the right pgbench database size scale] (archived article from 2009)<br />
<br />
==postgresql.conf parameters which need to be bumped up from defaults==<br />
* wal_buffers <br />
* shared_buffers<br />
* max_connections<br />
<br />
==Setup Tuning==<br />
For a relatively bigger setup, it is recommended to run pgbench client (the program pgbench itself) from a separate system than the system being tested. This would mean that network between the two system needs to be optimized. Monitor pgbench client itself to see that it does not become CPU saturated process (either via top or prstat -am) <br />
<br />
==Disk Tuning==<br />
* Separate out Log (pg_xlog) to different file system)<br />
<br />
== install note ==<br />
In the 9.1 package install for ubuntu pgbench is not wrapped for user, you can use the following to make it accessible<br />
<br />
sudo ln -s ../share/postgresql-common/pg_wrapper /usr/bin/pgbench<br />
<br />
[[Category:Benchmarking]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=32679Repmgr2018-10-25T02:51:03Z<p>Barwick: /* Downloads */ update link to package installation instructions</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL failover, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr helps DBAs and System Administrators manage a cluster of PostgreSQL databases. By taking advantage of the Hot Standby capability introduced in PostgreSQL 9, repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements.<br />
<br />
repmgr simplifies administration and daily management, enhances productivity and reduces the overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process<br />
* providing support for HA operations such as switch-overs and fail-overs.<br />
<br />
repmgr is available via 2ndQuadrant’s own package repositories as well as the PGDG community repositories.. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 4.x is compatible with all PostgreSQL versions from PostgreSQL 9.3 onwards.<br />
<br />
<br />
== repmgr 4 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/4.2/release-4.2.html 4.2] (2018-10-24).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or from Barman<br />
* Timeline following<br />
** a standby can be promoted to a primary without requiring a restart<br />
** other standbys can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the master to another standby mode<br />
* Replication slot support (PostgreSQL 9.4 and later), simplifying WAL retention management<br />
* [https://repmgr.org/docs/4.2/performing-switchover.html Switchover] support<br />
** supports role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at: https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Installation instructions for RedHat/Debian-based distributions are available [https://repmgr.org/docs/4.2/installation-packages.html here].<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=32678Repmgr2018-10-25T02:50:14Z<p>Barwick: repmgr 4.2 release</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL failover, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr helps DBAs and System Administrators manage a cluster of PostgreSQL databases. By taking advantage of the Hot Standby capability introduced in PostgreSQL 9, repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements.<br />
<br />
repmgr simplifies administration and daily management, enhances productivity and reduces the overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process<br />
* providing support for HA operations such as switch-overs and fail-overs.<br />
<br />
repmgr is available via 2ndQuadrant’s own package repositories as well as the PGDG community repositories.. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 4.x is compatible with all PostgreSQL versions from PostgreSQL 9.3 onwards.<br />
<br />
<br />
== repmgr 4 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/4.2/release-4.2.html 4.2] (2018-10-24).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or from Barman<br />
* Timeline following<br />
** a standby can be promoted to a primary without requiring a restart<br />
** other standbys can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the master to another standby mode<br />
* Replication slot support (PostgreSQL 9.4 and later), simplifying WAL retention management<br />
* [https://repmgr.org/docs/4.2/performing-switchover.html Switchover] support<br />
** supports role-switching between primary and standby<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at: https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Installation instructions for RedHat/Debian-based distributions are available [https://repmgr.org/docs/4.0/installation-packages.html here].<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=32075Repmgr2018-06-15T02:31:13Z<p>Barwick: Update repmgr details</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL failover, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr helps DBAs and System Administrators manage a cluster of PostgreSQL databases. By taking advantage of the Hot Standby capability introduced in PostgreSQL 9, repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements.<br />
<br />
repmgr simplifies administration and daily management, enhances productivity and reduces the overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process<br />
* providing support for HA operations such as switch-overs and fail-overs.<br />
<br />
repmgr is available via 2ndQuadrant’s own package repositories as well as the PGDG community repositories.. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 4.x is compatible with all PostgreSQL versions from PostgreSQL 9.3 onwards.<br />
<br />
<br />
== repmgr 4 Features ==<br />
<br />
'''Current release''': [https://repmgr.org/docs/4.0/release-4.0.6.html 4.0.6] (2018-06-14).<br />
<br />
* Implemented as a PostgreSQL extension<br />
* Replication cluster monitoring<br />
* Standby cloning with '''pg_basebackup''' or from Barman<br />
* Timeline following<br />
** a standby can be promoted to a primary without requiring a restart<br />
** other standbys can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the master to another standby mode<br />
* Replication slot support (PostgreSQL 9.4 and later), simplifying WAL retention management<br />
* [https://repmgr.org/docs/4.0/performing-switchover.html Switchover] support<br />
** supports role-switching between primary and standby<br />
<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at: https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [https://repmgr.org/ repmgr.org].<br />
<br />
Installation instructions for RedHat/Debian-based distributions are available [https://repmgr.org/docs/4.0/installation-packages.html here].<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Repmgr&diff=30399Repmgr2017-06-12T00:28:43Z<p>Barwick: Note PostgreSQL 10beta support in 3.3.2</p>
<hr />
<div>[https://2ndquadrant.com/en/resources/repmgr/ repmgr] is a popular tool for PostgreSQL failover, introduced by [https://2ndQuadrant.com 2ndQuadrant] in 2010. <br />
<br />
repmgr helps DBAs and System Administrators manage a cluster of PostgreSQL databases. By taking advantage of the Hot Standby capability introduced in PostgreSQL 9, repmgr greatly simplifies the process of setting up and managing databases with high availability (HA) and scalability requirements.<br />
<br />
repmgr simplifies administration and daily management, enhances productivity and reduces the overall costs of a PostgreSQL cluster by:<br />
<br />
* monitoring the replication process<br />
* providing support for HA operations such as switch-overs and fail-overs.<br />
<br />
repmgr is available via 2ndQuadrant’s YUM repository for the Red Hat family (RHEL, CentOS, and Fedora) and PGDG's APT repository for Debian. You can use standard yum and apt package managers for installing repmgr with your instance of PostgreSQL.<br />
<br />
* repmgr 3.x requires PostgreSQL 9.3 or higher<br />
* repmgr 2.x requires PostgreSQL 9.1 or 9.2<br />
<br />
== repmgr 3 Features ==<br />
<br />
'''Current release''': 3.3.2 (2017-06-08).<br />
<br />
* Barman support<br />
* Replication cluster monitoring<br />
* standby cloning with '''pg_basebackup''' or optionally '''rsync'''<br />
* Timeline following<br />
** a standby can be promoted to a master without requiring a restart<br />
** other standbys can connect to the new master without being resynced<br />
* Cascading standby support<br />
** Standbys not directly connected to the master node are not affected during failover of the master to another standby mode<br />
* Replication slot support (PostgreSQL 9.4 and later), simplifying WAL retention management<br />
* Switchover support<br />
** supports role-switching between master and standby<br />
* PostgreSQL 10beta support<br />
** Release 3.3.2 supports PostgreSQL 10beta for testing purposes<br />
<br />
== Downloads ==<br />
<br />
The repmgr project is hosted at: https://github.com/2ndQuadrant/repmgr<br />
<br />
Source code downloads are available from [http://www.repmgr.org/ www.repmgr.org].<br />
<br />
Installation instructions for RedHat/Debian-based distributions are available [https://2ndquadrant.com/en/resources/repmgr/installation-instructions/ here].<br />
<br />
== Administrative code snippets ==<br />
<br />
* [[Repmgr cleanup trigger]]<br />
<br />
[[Category:Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Multimaster&diff=30326Multimaster2017-06-06T02:17:45Z<p>Barwick: Multiple synchronous standbys from 9.6</p>
<hr />
<div><br />
PostgreSQL supports unidirectional master-slave replication. Moreover it supports ''hot standby'' mode in which it is possible to execute read only queries at replicas. Replication can be either asynchronous either synchronous, but even in case of synchronous replication there is a time gap between master and replica, so client executing read-only query at replica may not see changes it has performed in previous transaction at master. Also, prior to [https://www.postgresql.org/docs/current/static/release-9-6.html PostgreSQL 9.6] only one synchronous replica was supported, so it was practically not possible to use synchronous replication for load balancing. As a result, current streaming replication in PostgreSQL provides only fault tolerance (HA), but not scaling performance.<br />
<br />
[https://www.2ndquadrant.com/en/ 2ndQuadrant] provides [http://bdr-project.org/docs/stable/ Bidirectional Replication for PostgreSQL (BDR)]. In this case updates can be performed at any node of the cluster and then propagated to other nodes. BDR is essentially asynchronous: changes are applied at nodes some time later after committing transaction at master and various ways of resolving conflicts are proposed. BDR is really fast (provides almost the same speed as hot standby), but certainly there is no global consistency in such model.<br />
<br />
BDR is based on new PostgreSQL feature named [https://www.postgresql.org/docs/current/static/logicaldecoding-explanation.html logical decoding]. Changes are extracted from WAL and are proceeded by logical output plugin.<br />
In can then apply this changes to some other database, save in log or do whatever else it likes. BDR uses logical replication to deliver changes to other nodes. Logical replication is now part of PostgreSQL 9.5. Our multimaster is based on pglogical_output plugin provided by 2ndQuadrant. We have implemented receiver part for this plugin, which is also partly based on BDR code. At receiver side we have a pool of background workers which concurrently apply changes received from remote walsender. <br />
<br />
Global consistency is enforced by DTM (distributed transaction manager). From client's point of view it works just with set of identical PostgreSQL instances. It can login and send queries to any of them. It doesn't mean whether it is read-only or update query. But certainly, as far as updates has to be applied to all nodes, multimaster is able to provide scaling only for read-only queries.<br />
<br />
The diagram below shows performance results of multimaster installed at three nodes cluster. We run our dtmbench benchmark, varying percent of updates. We compare results of multimaster with performance of standalone PostgreSQL. Providing ACID properties for distributed transactions adds essential overhead: multimaster is about 2 times slower on updates than single node. In case of asynchronous replication it is possible to get much better results but without global consistency. At mostly read-only workloads multimaster provides much better performance, but still there is on linear scalability.<br />
<br />
<br />
[[File:rw_ratio.png]]<br />
<br />
[[File:reads.png]]<br />
<br />
[[File:updates.png]]<br />
<br />
Vertical axis: TPS, thouthands<br />
<br />
Horizontal axis: number of client connections</div>Barwickhttps://wiki.postgresql.org/index.php?title=Multimaster&diff=30325Multimaster2017-06-06T02:04:08Z<p>Barwick: Add relevant links</p>
<hr />
<div><br />
PostgreSQL supports unidirectional master-slave replication. Moreover it supports ''hot standby'' mode in which it is possible to execute read only queries at replicas. Replication can be either asynchronous either synchronous, but even in case of synchronous replication there is a time gap between master and replica, so client executing read-only query at replica may not see changes he has performed in previous transaction at master. Also, right now only one synchronous replica is supported, so it is practically not possible to use synchronous replication for load balancing. As a result, current streaming replication in PostgreSQL provides only fault tolerance (HA), but not scaling performance.<br />
<br />
[https://www.2ndquadrant.com/en/ 2ndQuadrant] provides [http://bdr-project.org/docs/stable/ Bidirectional Replication for PostgreSQL (BDR)]. In this case updates can be performed at any node of the cluster and then propagated to other nodes. BDR is essentially asynchronous: changes are applied at nodes some time later after committing transaction at master and various ways of resolving conflicts are proposed. BDR is really fast (provides almost the same speed as hot standby), but certainly there is no global consistency in such model.<br />
<br />
BDR is based on new PostgreSQL feature named [https://www.postgresql.org/docs/current/static/logicaldecoding-explanation.html logical decoding]. Changes are extracted from WAL and are proceeded by logical output plugin.<br />
In can then apply this changes to some other database, save in log or do whatever else it likes. BDR uses logical replication to deliver changes to other nodes. Logical replication is now part of PostgreSQL 9.5. Our multimaster is based on pglogical_output plugin provided by 2ndQuadrant. We have implemented receiver part for this plugin, which is also partly based on BDR code. At receiver side we have a pool of background workers which concurrently apply changes received from remote walsender. <br />
<br />
Global consistency is enforced by DTM (distributed transaction manager). From client's point of view it works just with set of identical PostgreSQL instances. It can login and send queries to any of them. It doesn't mean whether it is read-only or update query. But certainly, as far as updates has to be applied to all nodes, multimaster is able to provide scaling only for read-only queries.<br />
<br />
The diagram below shows performance results of multimaster installed at three nodes cluster. We run our dtmbench benchmark, varying percent of updates. We compare results of multimaster with performance of standalone PostgreSQL. Providing ACID properties for distributed transactions adds essential overhead: multimaster is about 2 times slower on updates than single node. In case of asynchronous replication it is possible to get much better results but without global consistency. At mostly read-only workloads multimaster provides much better performance, but still there is on linear scalability.<br />
<br />
<br />
[[File:rw_ratio.png]]<br />
<br />
[[File:reads.png]]<br />
<br />
[[File:updates.png]]<br />
<br />
Vertical axis: TPS, thouthands<br />
<br />
Horizontal axis: number of client connections</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_User_Guide&diff=30324BDR User Guide2017-06-06T01:43:51Z<p>Barwick: Update documentation links</p>
<hr />
<div>----<br />
'''This document has been superseded by the [http://bdr-project.org/docs/stable/index.html official BDR documentation].'''<br />
<br />
Other superseded BDR wiki pages redirect here. Click on the links below to access the relevant sections of the official BDR documentation.<br />
<br />
== BDR Quick Start Guide ==<br />
<br />
* [http://bdr-project.org/docs/stable/quickstart.html BDR Quick Start Guide]<br />
<br />
== Other links ==<br />
<br />
<br />
* [http://bdr-project.org/docs/stable/settings.html BDR Configuration Settings]<br />
* [http://bdr-project.org/docs/stable/ddl-replication-statements.html BDR DDL Replication and Issues]<br />
* [http://bdr-project.org/docs/stable/global-sequences.html BDR Global Sequences]<br />
* [http://bdr-project.org/docs/stable/functions.html BDR Functions]<br />
* [http://bdr-project.org/docs/stable/installation-packages.html BDR Installation from Packages]<br />
* [http://bdr-project.org/docs/stable/monitoring.html BDR Monitoring]<br />
* [http://bdr-project.org/docs/stable/conflicts.html BDR Multi-Master Conflicts]<br />
* [http://bdr-project.org/docs/stable/node-management.html BDR Node Management]<br />
* [http://bdr-project.org/docs/stable/overview.html BDR Overview and Comparison with Other Replication Methods]<br />
* [http://bdr-project.org/docs/stable/bdr-configuration-variables.html BDR Parameter Reference]<br />
* [http://bdr-project.org/docs/stable/catalogs-views.html BDR System Catalogues and Views]<br />
<br />
<br />
[[Category:Replication]][[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=Bi-Directional_Replication&diff=30323Bi-Directional Replication2017-06-06T01:41:36Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Quick_Start&diff=30322BDR Quick Start2017-06-06T01:40:22Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Command_Restrictions&diff=30321BDR Command Restrictions2017-06-06T01:39:28Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Catalogs&diff=30320BDR Catalogs2017-06-06T01:38:10Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Reference&diff=30319BDR Reference2017-06-06T01:36:32Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_User_Guide&diff=30318BDR User Guide2017-06-06T01:35:31Z<p>Barwick: Add direct links to BDR documentation for pages which redirect here</p>
<hr />
<div>----<br />
'''This document has been superseded by the [http://bdr-project.org/docs/stable/index.html official BDR documentation].'''<br />
<br />
Other superseded BDR wiki pages redirect here. Click on the links below to access the relevant sections of the BDR documentation.<br />
<br />
* [http://bdr-project.org/docs/stable/settings.html BDR Configuration Settings]<br />
* [http://bdr-project.org/docs/stable/global-sequences.html BDR Global Sequences]<br />
* [http://bdr-project.org/docs/stable/functions.html BDR Functions]<br />
* [http://bdr-project.org/docs/stable/installation-packages.html BDR Installation from Packages]<br />
* [http://bdr-project.org/docs/stable/monitoring.html BDR Monitoring]<br />
* [http://bdr-project.org/docs/stable/conflicts.html BDR Multi-Master Conflicts]<br />
* [http://bdr-project.org/docs/stable/node-management.html BDR Node Management]<br />
* [http://bdr-project.org/docs/stable/overview.html BDR Overview and Comparison with Other Replication Methods]<br />
* [http://bdr-project.org/docs/stable/bdr-configuration-variables.html BDR Parameter Reference]<br />
<br />
<br />
[[Category:Replication]][[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Parameter_Reference&diff=30317BDR Parameter Reference2017-06-06T01:34:37Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Packages&diff=30316BDR Packages2017-06-06T01:33:14Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Configurations&diff=30315BDR Configurations2017-06-06T01:31:58Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Comparisons&diff=30314BDR Comparisons2017-06-06T01:31:15Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Administration&diff=30313BDR Administration2017-06-06T01:29:13Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwickhttps://wiki.postgresql.org/index.php?title=BDR_Conflicts&diff=30312BDR Conflicts2017-06-06T01:27:54Z<p>Barwick: Redirect to BDR user guide (avoid having lots of stub pages hanging around)</p>
<hr />
<div>#REDIRECT [[BDR_User_Guide]]<br />
<br />
[[Category:Bi-Directional Replication]]</div>Barwick