https://wiki.postgresql.org/api.php?action=feedcontributions&user=Kaigai&feedformat=atomPostgreSQL wiki - User contributions [en]2024-03-29T07:16:30ZUser contributionsMediaWiki 1.35.13https://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=33357Foreign data wrappers2019-04-16T13:33:27Z<p>Kaigai: /* Exotic Wrappers */</p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained ?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|}<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/postgres-fdw.html documentation]<br />
|<br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
|<br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|FDW for MySQL<br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|Firebird<br />
|Native<br />
|<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|<br />
| currently work-in-progress.<br />
|-<br />
|SQLite<br />
|Native<br />
|<br />
|[https://github.com/gleu/sqlite_fdw github]<br />
|<br />
|<br />
|An FDW for SQLite3 (read-only)<br />
|-<br />
|[https://www.sqlite.org/index.html SQLite]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pgspider/sqlite_fdw github]<br />
|[https://pgxn.org/dist/sqlite_fdw PGXN]<br />
|[https://github.com/pgspider/sqlite_fdw/blob/master/README.md README]<br />
|An FDW for SQLite3 (write support and several pushdown optimization)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[https://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[https://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
| [https://github.com/griddb/griddb_nosql GridDB]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/griddb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/griddb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| InfluxDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/influxdb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/influxdb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|[https://multicorn.org/ Multicorn]<br />
|GPLv3<br />
|[https://github.com/sim51/neo4j-fdw Github]<br />
|<br />
|[https://github.com/sim51/neo4j-fdw/blob/master/README.adoc README]<br />
|FWD for Neo4j and also add a Cypher function to Pg<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/slamdata/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to]<br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| CSV / gzipped<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
|<br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
|<br />
|-<br />
| Multi-File<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| XML<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [https://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [https://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [https://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
|<br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
|<br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
|<br />
|}<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ICAL<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
|<br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
|<br />
|-<br />
| IMAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
|<br />
|-<br />
| RSS<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed.<br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Database.com<br />
| [https://multicorn.org/ Multicorn]<br />
| BSD<br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Dun & Badstreet<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
|<br />
|<br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Facebook<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
|<br />
| [https://github.com/hakanensari/frankfurter GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Keycloak<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/schne324/foreign-keycloak-wrapper GitHub]<br />
| [https://pgxn.org/dist/foreign-keycloak-wrapper/ PGXN]<br />
| [https://github.com/schne324/foreign-keycloak-wrapper/blob/master/README.md README]<br />
| Direct database integration with the [https://www.keycloak.org Keycloak] open-source Identity/Access Management solution.<br />
|-<br />
| Mailchimp<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
|<br />
|<br />
| Beta<br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| S3CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
|<br />
|<br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| Telegram<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/telegram_fdw GitHub]<br />
|<br />
|<br />
| telegram_fdw is a Telegram BOT implemented using the PostgreSQL foreign data wrapper interface.<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
|<br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
|<br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [https://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elastic Search<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/Mikulas/pg-es-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google BigQuery<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
|<br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
|<br />
|<br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://www.openscg.com/bigsql/hadoopfdw/ Bitbucket]<br />
|<br />
|<br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Hive<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
|<br />
|<br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://impala.apache.org/ Impala]<br />
| Native<br />
| BSD<br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://arrow.apache.org/ Apache Arrow]<br />
| Native<br />
| GPLv2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
|<br />
| A part of PG-Strom feature; as a columnar data source with support of SSD-to-GPU Direct SQL <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
|<br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
|<br />
|<br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Log files<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OpenStack / Telemetry<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OS Query<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
|<br />
|<br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
|<br />
|<br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
|<br />
|<br />
| A foreign datawrapper for querying system stats based on [https://libstatgrab.org/ statgrab]<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| faker_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/faker_fdw GitHub]<br />
|<br />
|<br />
| faker_fdw is a foreign data wrapper for PostgreSQL that generates fake data.<br />
|-<br />
| fdw_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
|<br />
|<br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PPG<br />
| Native<br />
|<br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
|<br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www2.meethue.com/en-us/philips-hue-benefits Phillips Hue Lighting Systems]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Random Number<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
|<br />
|<br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| [https://multicorn.org/ Multicorn]<br />
| Native<br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
| PostgreSQL<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator<br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
|<br />
|<br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
|<br />
|<br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Black Hole<br />
|<br />
|<br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
|<br />
|<br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [https://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=33356Foreign data wrappers2019-04-16T13:33:04Z<p>Kaigai: /* Big Data Wrappers */</p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+<br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained ?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|}<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/postgres-fdw.html documentation]<br />
|<br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
|<br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|FDW for MySQL<br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|Firebird<br />
|Native<br />
|<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|<br />
| currently work-in-progress.<br />
|-<br />
|SQLite<br />
|Native<br />
|<br />
|[https://github.com/gleu/sqlite_fdw github]<br />
|<br />
|<br />
|An FDW for SQLite3 (read-only)<br />
|-<br />
|[https://www.sqlite.org/index.html SQLite]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pgspider/sqlite_fdw github]<br />
|[https://pgxn.org/dist/sqlite_fdw PGXN]<br />
|[https://github.com/pgspider/sqlite_fdw/blob/master/README.md README]<br />
|An FDW for SQLite3 (write support and several pushdown optimization)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[https://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[https://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
| [https://github.com/griddb/griddb_nosql GridDB]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/griddb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/griddb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
| InfluxDB<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/pgspider/influxdb_fdw Github]<br />
|<br />
| [https://github.com/pgspider/influxdb_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[https://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|[https://multicorn.org/ Multicorn]<br />
|GPLv3<br />
|[https://github.com/sim51/neo4j-fdw Github]<br />
|<br />
|[https://github.com/sim51/neo4j-fdw/blob/master/README.adoc README]<br />
|FWD for Neo4j and also add a Cypher function to Pg<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/slamdata/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to]<br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| CSV / gzipped<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
|<br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
|<br />
|-<br />
| Multi-File<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| XML<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
|<br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [https://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [https://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [https://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF]<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
|<br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
|<br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
|<br />
|}<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ICAL<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
|<br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
|<br />
|-<br />
| IMAP<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
|<br />
|-<br />
| RSS<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [https://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed.<br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Database.com<br />
| [https://multicorn.org/ Multicorn]<br />
| BSD<br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Dun & Badstreet<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
|<br />
|<br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Facebook<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
|<br />
| [https://github.com/hakanensari/frankfurter GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
|<br />
|<br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Keycloak<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/schne324/foreign-keycloak-wrapper GitHub]<br />
| [https://pgxn.org/dist/foreign-keycloak-wrapper/ PGXN]<br />
| [https://github.com/schne324/foreign-keycloak-wrapper/blob/master/README.md README]<br />
| Direct database integration with the [https://www.keycloak.org Keycloak] open-source Identity/Access Management solution.<br />
|-<br />
| Mailchimp<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
|<br />
|<br />
| Beta<br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
|<br />
|<br />
|-<br />
| S3CSV<br />
| [https://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
|<br />
|<br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| Telegram<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/telegram_fdw GitHub]<br />
|<br />
|<br />
| telegram_fdw is a Telegram BOT implemented using the PostgreSQL foreign data wrapper interface.<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
|<br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
|<br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [https://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elastic Search<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/Mikulas/pg-es-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google BigQuery<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
|<br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
|<br />
|<br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://www.openscg.com/bigsql/hadoopfdw/ Bitbucket]<br />
|<br />
|<br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Hive<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
|<br />
|<br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://impala.apache.org/ Impala]<br />
| Native<br />
| BSD<br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://arrow.apache.org/ Apache Arrow]<br />
| Native<br />
| GPLv2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
|<br />
| A part of PG-Strom feature; as a columnar data source with support of SSD-to-GPU Direct SQL <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
|<br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [https://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
|<br />
|<br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Log files<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OpenStack / Telemetry<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| OS Query<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
|<br />
|<br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
|<br />
|<br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
|<br />
|<br />
| A foreign datawrapper for querying system stats based on [https://libstatgrab.org/ statgrab]<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| faker_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/guedes/faker_fdw GitHub]<br />
|<br />
|<br />
| faker_fdw is a foreign data wrapper for PostgreSQL that generates fake data.<br />
|-<br />
| fdw_fdw<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
|<br />
|<br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PGStrom<br />
| Native<br />
| GPL 2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
|<br />
| [http://wiki.postgresql.org/wiki/PGStrom wiki]<br />
| uses GPU devices to accelarate sequential scan on massive amount of records with complex qualifiers. Now it moved to CustomScan based implementation, so its core logic no longer uses FDW.<br />
|-<br />
| PPG<br />
| Native<br />
|<br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
|<br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [https://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www2.meethue.com/en-us/philips-hue-benefits Phillips Hue Lighting Systems]<br />
| [https://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Random Number<br />
| [https://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
|<br />
|<br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| [https://multicorn.org/ Multicorn]<br />
| Native<br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
| PostgreSQL<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator<br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
|<br />
|<br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |License<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
|<br />
|<br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
|<br />
|<br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Black Hole<br />
|<br />
|<br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
|<br />
|<br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [https://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2018_Developer_Unconference&diff=32881PGConf.ASIA2018 Developer Unconference2018-12-10T07:05:22Z<p>Kaigai: /* Partition-wise Join & Visibility Map */</p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 10 Dec, 2018 at Akihabara Convention Hall, as part of [http://www.pgconf.asia/EN/2018/ PGConf.ASIA 2018]. <br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Okinawa<br />
!Hokkaido<br />
|- style="background-color:lightgray;"<br />
|10:30-10:50<br />
|colspan="2"|Voting<br />
<br />
|-<br />
|11:00-11:50<br />
| PostgreSQL and Data Analytics<br />
| Referential Integrity Using Statement Level Triggers<br />
<br />
|- style="background-color:lightgray;"<br />
|11:50 - 13:00 <br />
|colspan="2"|Lunch<br />
<br />
|-<br />
|13:00-13:50 <br />
|Shared Buffer Manager<br />
|Hypotehtical Partitioning<br />
<br />
|-<br />
|14:00-14:50 <br />
|Partition-wise JOIN and Visibility Map<br />
|Query Monitoring<br />
<br />
|-<br />
|15:00-15:50<br />
|Shared Catalog and Relation Cache<br />
|Built In Job Scheduler<br />
<br />
|-<br />
|16:00-16:50<br />
|Per-tablespace transparent encryption<br />
|Idle Connection Scaling<br />
<br />
|- style="background-color:lightgray;"<br />
|17:00 - <br />
|colspan="2"|Beer Elsewhere<br />
<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
* TBD<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
<br />
=== Statement level referential integrity ===<br />
<br />
Oracle-style import<br />
No way to invalidate then revalidate constraints<br />
Right thing also the last thing<br />
Checks fire once per row<br />
Kevin Grittner + Thomas Munro: statement-level triggers, transition tables<br />
benchmark: 98.x% faster<br />
no code, just a PL/PgSQL script<br />
REFERENCES: INSERT/UPDATE trigger on child, DELETE trigger on parent<br />
initial pass: LEFT OUTER JOIN<br />
code is already there!<br />
transition tables don't necessarily have names<br />
enforce constraints with a single table scan! important when _appending_ to a table<br />
* CreateFKCheckTrigger<br />
* SPI_register_triger_data<br />
very seldom more statements than rows<br />
issues with the error message: just show the first row? doesn't make things worse<br />
ignore failed rows server-side; report them so app can fix them (different thing)<br />
different rejection tables for every failure?<br />
Kevin Grittner: delta relations in ALTER triggers [pgsql-hackers]<br />
* commands/tablecmds.c<br />
* RI_Initial_Check<br />
are statement-level triggers also `Trigger *`?<br />
what does the benchmark look like for one-row-at-a-time? make the row count tunable?<br />
how to upgrade? find all row-level triggers and replace them<br />
not a concern with dump/restore, but pg_upgrade needs to skip the initial check<br />
triggers have PG-reserved names; use that to detect and upgrade triggers<br />
pg_upgrade creates empty cluster; maybe no need to skip check?<br />
all that's missing is the magic to convert the trigger<br />
no catalogs are binary-linked<br />
pg_dump/pg_restore dump standard SQL; no syntax changes<br />
switch from row-level to statement-level after X rows<br />
parallel query uses a queue already<br />
suggestion: have custom non-trigger mechanism<br />
<br />
=== Query Monitoring (enhancing pg_stat_statements) ===<br />
<br />
Common complaints:<br />
<br />
* cannot see actual values used in specific queries<br />
* cannot see the query plan that was used<br />
<br />
pg_stat_statements today<br />
<br />
* dbid<br />
* userid<br />
* queryid<br />
* query<br />
<br />
Brief explanation of pg_qualstats and how it works<br />
<br />
* high overhead to deparsing<br />
* lots of quals can be slow<br />
* sampling rate of queries deparsed is < 5%<br />
<br />
Mention of pg_stat_plans and pg_hint_plans and <br />
<br />
[https://github.com/ossc-db/pg_store_plans pg_store_plans]<br />
<br />
* also uses query id<br />
* unclear LICENSE (issue opened)<br />
<br />
Issue with auto_explain, no way to link it back to pg_stat_statements<br />
<br />
Issue with how search_path can alter queries and cause pg_stat_statements to conflate unrelated but textually similar queries<br />
<br />
Discussion of query performance variance and storing historgrams, and the limitations of those histograms.<br />
<br />
Discussion of whether it would benefit us to have indicators of which queries had the most sequential scans, index scans, etc.<br />
<br />
pg_stat_kcache, discussion of its utility, ability to gather OS-level CPU usage, I/O usage for the life of the session, which with the right sampling can be interpolated to the per-query level.<br />
<br />
Discussion of tracking wait events. <br />
<br />
Should queryid be in pg_stat_activity? PostgresPro already has an extension that does this: [https://github.com/postgrespro/pg_wait_sampling pg_wait_sampling].<br />
<br />
Issue: canceled queries don't show up in pg_stat_statements.<br />
<br />
=== PostgreSQL and Data Analytics ===<br />
<br />
Karsten: Perspective from the user<br />
<br />
1) Level of parallelism the query planner is creating is not optimal - Postgres does not create enough parallelism<br><br />
2) Customers find database crashing when number of connections goes up (out of memory errors)<br />
<br />
=> You can tune the Linux to avoid crashes and instead return out of memory errors (tell Linux to not overcommit memory)<br />
<br />
Joe Conway: "What is many connections?"<br><br />
Karsten: 1000 connections<br />
<br />
Joe Conway: Thats a lot of connections, in particular for large analytics workloads<br><br />
Joe Conway: You have to shrink work_mem when you use that many connections<br />
<br />
Lukas Fittl: Is it useful to look at work_mem as a database-wide setting instead of per-connection? (there is an existing -hackers thread on this)<br><br />
Joe Conway: People have talked about resource management over the years, its one of the features that Postgres lacks that other databases have<br />
<br />
---<br />
<br />
Karsten: To the other topic, not enough parallelism - how could parallelism work better?<br><br />
Joe Conway: What are you bottlenecked on, I/O?<br><br />
Karsten: The data is cached, in memory, you have a database of 100GB of memory, and still its not working in parallel<br><br />
Karsten: Seeing this same problem when running TPC-H benchmarks<br><br />
(?): Would need to look at the specific query execution plan to understand the parallelism problems better<br><br />
(?): Memory usage is shared_buffers, per connection relation cache, etc. - not just work_mem<br />
<br />
---<br />
<br />
Joe Conway: Do we need per-user memory restrictions?<br><br />
Karsten: In our case, not so much, its typically one user running the queries<br><br />
Karsten: In our environments, queries can easily run for 30, 60 minutes, or longer<br />
<br />
Michael Paquier: There is a memory debug function in Postgres (the same that prints in OOM situations), that could be utilized<br><br />
Joe Conway: Someone needs to sit down and come up with a list of requirements on resource management<br />
<br />
Michael Paquier: One idea could be to have a per-session memory limit that is managed by each backend<br><br />
Joe Conway: Whatever you do in this area, it would have to not be overrideable per-user (from a security perspective)<br />
<br />
[ discussion around measuring whether parallel query works ]<br />
<br />
(?): There are two counters in shared memory around parallel workers launched/etc that could be exported<br />
<br />
<br />
=== Hypothetical Partitions for testing optimizations ===<br />
<br />
- like hypopg does for indexes, but with partitions<br />
<br />
challenges:<br />
- how to get a fake table name, and how this is already accomplished for index relations in hypopg<br />
- relcache does not have the ability to remove negative relcache entries<br />
- how to impose catalog entries to represent the partitions in just one session<br />
- velocity of changes to partitioning in recent versions (10,11, master) means that few people understand what current behavior is<br />
- some insert/delete hooks coming in v12, which may cut down the amount of code that needs to be duplicated<br />
- CreateFakeRelcacheEntry / FreeFakeRelcacheEntry<br />
- How to select Oids for fake objects on a read replica, risk collision with new real objects from master. Some question of whether fake Oids a<br />
re even necessary.<br />
<br />
It was suggested that refactoring for v12, even if incomplete, is better than no progress at all.<br />
<br />
Several user questions about hypopg and how stable it is, and how it would be used, with an eye toward including in cloud DBAAS products.<br />
<br />
Some concern was expressed that it may be too late in the development cycle to get a chance of this size into v12.<br />
<br />
=== Partition-wise Join & Visibility Map ===<br />
<br />
KaiGai Kohei (HeteroDB,Inc)<br />
<br />
First, KaiGai introduced the background why he focuses on the features.<br />
<br />
He has developed PG-Strom which is an extension of PostgreSQL to accelerate analytic queries using GPU. It also challenges to I/O intensive workloads by its SSD-to-GPU Direct SQL feature; that loads PostgreSQL's data blocks on the storage (NVME-SSD) to GPU directly using P2P DMA, then GPU runs a part of SQL workloads (WHERE/JOIN/GROUP BY). It reduces the data size to be loaded to CPU/RAM.<br />
<br />
Some HPC hardware supports PCIe switch between CPU and peripheral devices on PCIe bus. It also enables to bypass PCIe controller built-in CPU for SSD-to-GPU P2P DMA, if a pair of SSD and GPU are installed under the same PCIe-switch.<br />
From the standpoint of PostgreSQL, it makes sense that partition leaf accommodates bunch of SSDs for each PCIe-switch.<br />
Current version of partition-wise join allows JOIN between partitioned-tables with same distributed key, however, it makes sense to distribute and push down normal tables to individual partition-leafs, because we can run SQL on the peripheral device side.<br />
<br />
He suggested, 1) a lightweight hook on SetHintBit() to manage the heap blocks which are safe for P2P direct read, because GPU cannot access commit log, and zHeap may eventually remove the visibility-map infrastructure.<br />
2) an infrastructure to make a join path between normal-table and partitioned-table, without redundant inner read.<br />
<br />
From the audience, they comment on materialization is valuable for merge join also, not only hash-join. Even if inner table is distributed to 100 partition-leaf, shared materialized image prevents repeat of same scan. One other also comments the hashed/materialized table can be reduced if join-key contains partition-key and we can skip tuples obviously unmatch. One asked it is safe for non-equality join. KaiGai thinks it is equivalent to the current behavior for INNER JOIN, even if reorder GATHER and JOIN.<br />
One other question was about the hook on SetHintBit() - which information shall be delivered. KaiGai said that relation-id, block-number, iterm-pointer and informask, but it is detailed stuff.<br />
<br />
session slides are here:<br />
https://www.slideshare.net/kaigai/20181210-pgconfasia-unconference<br />
<br />
=== Built-In Job Scheduler ===<br />
<br />
Lack of scheduler in Postgres<br />
<br />
Extensions end up making their own scheduler, or require people to use cron<br />
<br />
Use cases<br />
<br />
* Partition rotation (pg_partman, Timescale)<br />
* Run-away queries and killing them<br />
* Time-based partial indexes (queries almost always look for the last one of something)<br />
* Manual vacuums<br />
<br />
In-core scheduler can also be useful for extensions to add recurring tasks, no need to make their own background worker<br />
<br />
Key features<br />
- RLS (Row Level Security) on user_name<br />
- Do NOT look like cron<br />
- Only execute stored procedures with a specific call signature<br />
<br />
We have stored procedures - no need for another program!<br />
<br />
<pre><br />
pg_job_schedule<br />
- job_name text UNIQUE<br />
- user_name text<br />
- procedure_name text/oid<br />
- initial_run tstz<br />
- run_frequency interval<br />
- last_run tstz<br />
- last_duration interval<br />
- last_successful_run tstz<br />
- last_successful_duration interval<br />
- max_run_duration interval<br />
- num_overlap int<br />
</pre><br />
<br />
Stored procedures are currently restricted in terms of languages, but from there you could call anything<br />
<br />
"How many backends do you use?"<br><br />
=> One scheduler job (background worker) that would then spawn separate workers for each invocation<br />
<br />
Dimitri: Should have a cap on the number of concurrent tasks that can be started<br />
<br />
Magnus: If you want long-running jobs, you might want to put priority on them<br />
<br />
Flag to run things on a primary/standby only (could also be solved with pg_is_in_recovery())<br />
<br />
What wouldn't work with this?<br><br />
=> VACUUM doesn't work, because it checks whether its a top-level statement<br><br />
=> CREATE INDEX CONCURRENTLY<br />
<br />
"Whats the value to only restricting it to execute stored procedures?"<br><br />
=> Simplicity was the main motivation<br><br />
=> One command text field is good too<br />
<br />
"Which databases is it going to run on?"<br><br />
=> Would need a way to specify the database<br />
<br />
"Are timezones a problem - how do you handle a day-light savings time change?"<br><br />
=> This should be taken care of by the schema as proposed<br />
<br />
"Isn't this similar to the ability to write a background worker without using C?"<br />
<br />
Magnus: "I want the ability to run a job now, without any other effects on the schedule"<br />
<br />
"Are last_duration / last_successful_run / etc necessary at all / should they be in a separate table?"<br><br />
=> Maybe should have "pg_stat_jobs_schedule" to store these and other things like rusage<br />
<br />
The core feature that could be using this is continuous materialized views<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2016_Developer_Unconference PgConf.ASIA 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2017_Developer_Unconference PgCon 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2017_Developer_Unconference PgConf.ASIA 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2018_Developer_Unconference PgCon 2018 Developer Unconference]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2018_Developer_Unconference&diff=32880PGConf.ASIA2018 Developer Unconference2018-12-10T07:01:29Z<p>Kaigai: /* Partition-wise Join & Visibility Map */</p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 10 Dec, 2018 at Akihabara Convention Hall, as part of [http://www.pgconf.asia/EN/2018/ PGConf.ASIA 2018]. <br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Okinawa<br />
!Hokkaido<br />
|- style="background-color:lightgray;"<br />
|10:30-10:50<br />
|colspan="2"|Voting<br />
<br />
|-<br />
|11:00-11:50<br />
| PostgreSQL and Data Analytics<br />
| Referential Integrity Using Statement Level Triggers<br />
<br />
|- style="background-color:lightgray;"<br />
|11:50 - 13:00 <br />
|colspan="2"|Lunch<br />
<br />
|-<br />
|13:00-13:50 <br />
|Shared Buffer Manager<br />
|Hypotehtical Partitioning<br />
<br />
|-<br />
|14:00-14:50 <br />
|Partition-wise JOIN and Visibility Map<br />
|Query Monitoring<br />
<br />
|-<br />
|15:00-15:50<br />
|Shared Catalog and Relation Cache<br />
|Built In Job Scheduler<br />
<br />
|-<br />
|16:00-16:50<br />
|Per-tablespace transparent encryption<br />
|Idle Connection Scaling<br />
<br />
|- style="background-color:lightgray;"<br />
|17:00 - <br />
|colspan="2"|Beer Elsewhere<br />
<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
* TBD<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
<br />
=== Statement level referential integrity ===<br />
<br />
Oracle-style import<br />
No way to invalidate then revalidate constraints<br />
Right thing also the last thing<br />
Checks fire once per row<br />
Kevin Grittner + Thomas Munro: statement-level triggers, transition tables<br />
benchmark: 98.x% faster<br />
no code, just a PL/PgSQL script<br />
REFERENCES: INSERT/UPDATE trigger on child, DELETE trigger on parent<br />
initial pass: LEFT OUTER JOIN<br />
code is already there!<br />
transition tables don't necessarily have names<br />
enforce constraints with a single table scan! important when _appending_ to a table<br />
* CreateFKCheckTrigger<br />
* SPI_register_triger_data<br />
very seldom more statements than rows<br />
issues with the error message: just show the first row? doesn't make things worse<br />
ignore failed rows server-side; report them so app can fix them (different thing)<br />
different rejection tables for every failure?<br />
Kevin Grittner: delta relations in ALTER triggers [pgsql-hackers]<br />
* commands/tablecmds.c<br />
* RI_Initial_Check<br />
are statement-level triggers also `Trigger *`?<br />
what does the benchmark look like for one-row-at-a-time? make the row count tunable?<br />
how to upgrade? find all row-level triggers and replace them<br />
not a concern with dump/restore, but pg_upgrade needs to skip the initial check<br />
triggers have PG-reserved names; use that to detect and upgrade triggers<br />
pg_upgrade creates empty cluster; maybe no need to skip check?<br />
all that's missing is the magic to convert the trigger<br />
no catalogs are binary-linked<br />
pg_dump/pg_restore dump standard SQL; no syntax changes<br />
switch from row-level to statement-level after X rows<br />
parallel query uses a queue already<br />
suggestion: have custom non-trigger mechanism<br />
<br />
=== Query Monitoring (enhancing pg_stat_statements) ===<br />
<br />
Common complaints:<br />
<br />
* cannot see actual values used in specific queries<br />
* cannot see the query plan that was used<br />
<br />
pg_stat_statements today<br />
<br />
* dbid<br />
* userid<br />
* queryid<br />
* query<br />
<br />
Brief explanation of pg_qualstats and how it works<br />
<br />
* high overhead to deparsing<br />
* lots of quals can be slow<br />
* sampling rate of queries deparsed is < 5%<br />
<br />
Mention of pg_stat_plans and pg_hint_plans and <br />
<br />
[https://github.com/ossc-db/pg_store_plans pg_store_plans]<br />
<br />
* also uses query id<br />
* unclear LICENSE (issue opened)<br />
<br />
Issue with auto_explain, no way to link it back to pg_stat_statements<br />
<br />
Issue with how search_path can alter queries and cause pg_stat_statements to conflate unrelated but textually similar queries<br />
<br />
Discussion of query performance variance and storing historgrams, and the limitations of those histograms.<br />
<br />
Discussion of whether it would benefit us to have indicators of which queries had the most sequential scans, index scans, etc.<br />
<br />
pg_stat_kcache, discussion of its utility, ability to gather OS-level CPU usage, I/O usage for the life of the session, which with the right sampling can be interpolated to the per-query level.<br />
<br />
Discussion of tracking wait events. <br />
<br />
Should queryid be in pg_stat_activity? PostgresPro already has an extension that does this: [https://github.com/postgrespro/pg_wait_sampling pg_wait_sampling].<br />
<br />
Issue: canceled queries don't show up in pg_stat_statements.<br />
<br />
=== PostgreSQL and Data Analytics ===<br />
<br />
Karsten: Perspective from the user<br />
<br />
1) Level of parallelism the query planner is creating is not optimal - Postgres does not create enough parallelism<br><br />
2) Customers find database crashing when number of connections goes up (out of memory errors)<br />
<br />
=> You can tune the Linux to avoid crashes and instead return out of memory errors (tell Linux to not overcommit memory)<br />
<br />
Joe Conway: "What is many connections?"<br><br />
Karsten: 1000 connections<br />
<br />
Joe Conway: Thats a lot of connections, in particular for large analytics workloads<br><br />
Joe Conway: You have to shrink work_mem when you use that many connections<br />
<br />
Lukas Fittl: Is it useful to look at work_mem as a database-wide setting instead of per-connection? (there is an existing -hackers thread on this)<br><br />
Joe Conway: People have talked about resource management over the years, its one of the features that Postgres lacks that other databases have<br />
<br />
---<br />
<br />
Karsten: To the other topic, not enough parallelism - how could parallelism work better?<br><br />
Joe Conway: What are you bottlenecked on, I/O?<br><br />
Karsten: The data is cached, in memory, you have a database of 100GB of memory, and still its not working in parallel<br><br />
Karsten: Seeing this same problem when running TPC-H benchmarks<br><br />
(?): Would need to look at the specific query execution plan to understand the parallelism problems better<br><br />
(?): Memory usage is shared_buffers, per connection relation cache, etc. - not just work_mem<br />
<br />
---<br />
<br />
Joe Conway: Do we need per-user memory restrictions?<br><br />
Karsten: In our case, not so much, its typically one user running the queries<br><br />
Karsten: In our environments, queries can easily run for 30, 60 minutes, or longer<br />
<br />
Michael Paquier: There is a memory debug function in Postgres (the same that prints in OOM situations), that could be utilized<br><br />
Joe Conway: Someone needs to sit down and come up with a list of requirements on resource management<br />
<br />
Michael Paquier: One idea could be to have a per-session memory limit that is managed by each backend<br><br />
Joe Conway: Whatever you do in this area, it would have to not be overrideable per-user (from a security perspective)<br />
<br />
[ discussion around measuring whether parallel query works ]<br />
<br />
(?): There are two counters in shared memory around parallel workers launched/etc that could be exported<br />
<br />
<br />
=== Hypothetical Partitions for testing optimizations ===<br />
<br />
- like hypopg does for indexes, but with partitions<br />
<br />
challenges:<br />
- how to get a fake table name, and how this is already accomplished for index relations in hypopg<br />
- relcache does not have the ability to remove negative relcache entries<br />
- how to impose catalog entries to represent the partitions in just one session<br />
- velocity of changes to partitioning in recent versions (10,11, master) means that few people understand what current behavior is<br />
- some insert/delete hooks coming in v12, which may cut down the amount of code that needs to be duplicated<br />
- CreateFakeRelcacheEntry / FreeFakeRelcacheEntry<br />
- How to select Oids for fake objects on a read replica, risk collision with new real objects from master. Some question of whether fake Oids a<br />
re even necessary.<br />
<br />
It was suggested that refactoring for v12, even if incomplete, is better than no progress at all.<br />
<br />
Several user questions about hypopg and how stable it is, and how it would be used, with an eye toward including in cloud DBAAS products.<br />
<br />
Some concern was expressed that it may be too late in the development cycle to get a chance of this size into v12.<br />
<br />
=== Partition-wise Join & Visibility Map ===<br />
<br />
KaiGai Kohei (HeteroDB,Inc)<br />
<br />
First, KaiGai introduced the background why he focuses on the features.<br />
<br />
He has developed PG-Strom which is an extension of PostgreSQL to accelerate analytic queries using GPU. It also challenges to I/O intensive workloads by its SSD-to-GPU Direct SQL feature; that loads PostgreSQL's data blocks on the storage (NVME-SSD) to GPU directly using P2P DMA, then GPU runs a part of SQL workloads (WHERE/JOIN/GROUP BY). It reduces the data size to be loaded to CPU/RAM.<br />
<br />
Some HPC hardware supports PCIe switch between CPU and peripheral devices on PCIe bus. It also enables to bypass PCIe controller built-in CPU for SSD-to-GPU P2P DMA, if a pair of SSD and GPU are installed under the same PCIe-switch.<br />
From the standpoint of PostgreSQL, it makes sense that partition leaf accommodates bunch of SSDs for each PCIe-switch.<br />
Current version of partition-wise join allows JOIN between partitioned-tables with same distributed key, however, it makes sense to distribute and push down normal tables to individual partition-leafs, because we can run SQL on the peripheral device side.<br />
<br />
He suggested, 1) a lightweight hook on SetHintBit() to manage the heap blocks which are safe for P2P direct read, because GPU cannot access commit log, and zHeap may eventually remove the visibility-map infrastructure.<br />
2) an infrastructure to make a join path between normal-table and partitioned-table, without redundant inner read.<br />
<br />
From the audience, they comment on materialization is valuable for merge join also, not only hash-join. Even if inner table is distributed to 100 partition-leaf, shared materialized image prevents repeat of same scan. One other also comments the hashed/materialized table can be reduced if join-key contains partition-key and we can skip tuples obviously unmatch. One asked it is safe for non-equality join. KaiGai thinks it is equivalent to the current behavior for INNER JOIN, even if reorder GATHER and JOIN.<br />
One other question was about the hook on SetHintBit() - which information shall be delivered. KaiGai said that relation-id, block-number, iterm-pointer and informask, but it is detailed stuff.<br />
<br />
=== Built-In Job Scheduler ===<br />
<br />
Lack of scheduler in Postgres<br />
<br />
Extensions end up making their own scheduler, or require people to use cron<br />
<br />
Use cases<br />
<br />
* Partition rotation (pg_partman, Timescale)<br />
* Run-away queries and killing them<br />
* Time-based partial indexes (queries almost always look for the last one of something)<br />
* Manual vacuums<br />
<br />
In-core scheduler can also be useful for extensions to add recurring tasks, no need to make their own background worker<br />
<br />
Key features<br />
- RLS (Row Level Security) on user_name<br />
- Do NOT look like cron<br />
- Only execute stored procedures with a specific call signature<br />
<br />
We have stored procedures - no need for another program!<br />
<br />
<pre><br />
pg_job_schedule<br />
- job_name text UNIQUE<br />
- user_name text<br />
- procedure_name text/oid<br />
- initial_run tstz<br />
- run_frequency interval<br />
- last_run tstz<br />
- last_duration interval<br />
- last_successful_run tstz<br />
- last_successful_duration interval<br />
- max_run_duration interval<br />
- num_overlap int<br />
</pre><br />
<br />
Stored procedures are currently restricted in terms of languages, but from there you could call anything<br />
<br />
"How many backends do you use?"<br><br />
=> One scheduler job (background worker) that would then spawn separate workers for each invocation<br />
<br />
Dimitri: Should have a cap on the number of concurrent tasks that can be started<br />
<br />
Magnus: If you want long-running jobs, you might want to put priority on them<br />
<br />
Flag to run things on a primary/standby only (could also be solved with pg_is_in_recovery())<br />
<br />
What wouldn't work with this?<br><br />
=> VACUUM doesn't work, because it checks whether its a top-level statement<br><br />
=> CREATE INDEX CONCURRENTLY<br />
<br />
"Whats the value to only restricting it to execute stored procedures?"<br><br />
=> Simplicity was the main motivation<br><br />
=> One command text field is good too<br />
<br />
"Which databases is it going to run on?"<br><br />
=> Would need a way to specify the database<br />
<br />
"Are timezones a problem - how do you handle a day-light savings time change?"<br><br />
=> This should be taken care of by the schema as proposed<br />
<br />
"Isn't this similar to the ability to write a background worker without using C?"<br />
<br />
Magnus: "I want the ability to run a job now, without any other effects on the schedule"<br />
<br />
"Are last_duration / last_successful_run / etc necessary at all / should they be in a separate table?"<br><br />
=> Maybe should have "pg_stat_jobs_schedule" to store these and other things like rusage<br />
<br />
The core feature that could be using this is continuous materialized views<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2016_Developer_Unconference PgConf.ASIA 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2017_Developer_Unconference PgCon 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2017_Developer_Unconference PgConf.ASIA 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2018_Developer_Unconference PgCon 2018 Developer Unconference]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2018_Developer_Unconference&diff=32875PGConf.ASIA2018 Developer Unconference2018-12-10T06:23:40Z<p>Kaigai: /* Hypothetical Partitions for testing optimizations */</p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 10 Dec, 2018 at Akihabara Convention Hall, as part of [http://www.pgconf.asia/EN/2018/ PGConf.ASIA 2018]. <br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Okinawa<br />
!Hokkaido<br />
|- style="background-color:lightgray;"<br />
|10:30-10:50<br />
|colspan="2"|Voting<br />
<br />
|-<br />
|11:00-11:50<br />
| PostgreSQL and Data Analytics<br />
| Referential Integrity Using Statement Level Triggers<br />
<br />
|- style="background-color:lightgray;"<br />
|11:50 - 13:00 <br />
|colspan="2"|Lunch<br />
<br />
|-<br />
|13:00-13:50 <br />
|Shared Buffer Manager<br />
|Hypotehtical Partitioning<br />
<br />
|-<br />
|14:00-14:50 <br />
|Partition-wise JOIN and Visibility Map<br />
|Query Monitoring<br />
<br />
|-<br />
|15:00-15:50<br />
|Shared Catalog and Relation Cache<br />
|Built In Job Scheduler<br />
<br />
|-<br />
|16:00-16:50<br />
|Per-tablespace transparent encryption<br />
|Idle Connection Scaling<br />
<br />
|- style="background-color:lightgray;"<br />
|17:00 - <br />
|colspan="2"|Beer Elsewhere<br />
<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
* TBD<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
<br />
=== Statement level referential integrity ===<br />
<br />
Oracle-style import<br />
No way to invalidate then revalidate constraints<br />
Right thing also the last thing<br />
Checks fire once per row<br />
Kevin Grittner + Thomas Munro: statement-level triggers, transition tables<br />
benchmark: 98.x% faster<br />
no code, just a PL/PgSQL script<br />
REFERENCES: INSERT/UPDATE trigger on child, DELETE trigger on parent<br />
initial pass: LEFT OUTER JOIN<br />
code is already there!<br />
transition tables don't necessarily have names<br />
enforce constraints with a single table scan! important when _appending_ to a table<br />
* CreateFKCheckTrigger<br />
* SPI_register_triger_data<br />
very seldom more statements than rows<br />
issues with the error message: just show the first row? doesn't make things worse<br />
ignore failed rows server-side; report them so app can fix them (different thing)<br />
different rejection tables for every failure?<br />
Kevin Grittner: delta relations in ALTER triggers [pgsql-hackers]<br />
* commands/tablecmds.c<br />
* RI_Initial_Check<br />
are statement-level triggers also `Trigger *`?<br />
what does the benchmark look like for one-row-at-a-time? make the row count tunable?<br />
how to upgrade? find all row-level triggers and replace them<br />
not a concern with dump/restore, but pg_upgrade needs to skip the initial check<br />
triggers have PG-reserved names; use that to detect and upgrade triggers<br />
pg_upgrade creates empty cluster; maybe no need to skip check?<br />
all that's missing is the magic to convert the trigger<br />
no catalogs are binary-linked<br />
pg_dump/pg_restore dump standard SQL; no syntax changes<br />
switch from row-level to statement-level after X rows<br />
parallel query uses a queue already<br />
suggestion: have custom non-trigger mechanism<br />
<br />
=== Query Monitoring (enhancing pg_stat_statements) ===<br />
<br />
Common complaints:<br />
- cannot see actual values used in specific queries<br />
- cannot see the query plan that was used<br />
<br />
<br />
What fields would be needed to track specific parameters for queries<br />
- dbid<br />
- userid<br />
- queryid<br />
- query<br />
- constants<br />
- parameters<br />
<br />
Brief explanation of pg_qualstats and how it works<br />
- high overhead to deparsing<br />
- lots of quals can be slow<br />
- sampling rate of queries deparsed is < 5%<br />
<br />
Mention of pg_stat_plans and pg_hint_plans and <br />
<br />
pg_store_plans<br />
- also uses query id<br />
<br />
Issue with auto_explain, no way to link it back to pg_stat_statements<br />
<br />
Issue with how search_path can alter queries and cause pg_stat_statements to conflate unrelated but textually similar queries<br />
<br />
Discussion of query performance variance and storing historgrams, and the limitations of those histograms.<br />
<br />
Discussion of whether it would benefit us to have indicators of which queries had the most sequential scans, index scans, etc.<br />
<br />
pg_stat_kcache, discussion of its utility, ability to gather OS-level CPU usage, I/O usage for the life of the session, which with the right sa<br />
mpling can be interpolated to the per-query level.<br />
<br />
<br />
Discussion of tracking wait events. <br />
<br />
Should queryid be in pg_stat_activity? PostgresPro already has an extension that does this.<br />
<br />
Issue: canceled queries don't show up in pg_stat_statements.<br />
<br />
=== PostgreSQL and Data Analytics ===<br />
<br />
Karsten: Perspective from the user<br />
<br />
1) Level of parallelism the query planner is creating is not optimal - Postgres does not create enough parallelism<br><br />
2) Customers find database crashing when number of connections goes up (out of memory errors)<br />
<br />
=> You can tune the Linux to avoid crashes and instead return out of memory errors (tell Linux to not overcommit memory)<br />
<br />
Joe Conway: "What is many connections?"<br><br />
Karsten: 1000 connections<br />
<br />
Joe Conway: Thats a lot of connections, in particular for large analytics workloads<br><br />
Joe Conway: You have to shrink work_mem when you use that many connections<br />
<br />
Lukas Fittl: Is it useful to look at work_mem as a database-wide setting instead of per-connection? (there is an existing -hackers thread on this)<br><br />
Joe Conway: People have talked about resource management over the years, its one of the features that Postgres lacks that other databases have<br />
<br />
---<br />
<br />
Karsten: To the other topic, not enough parallelism - how could parallelism work better?<br><br />
Joe Conway: What are you bottlenecked on, I/O?<br><br />
Karsten: The data is cached, in memory, you have a database of 100GB of memory, and still its not working in parallel<br><br />
Karsten: Seeing this same problem when running TPC-H benchmarks<br><br />
(?): Would need to look at the specific query execution plan to understand the parallelism problems better<br><br />
(?): Memory usage is shared_buffers, per connection relation cache, etc. - not just work_mem<br />
<br />
---<br />
<br />
Joe Conway: Do we need per-user memory restrictions?<br><br />
Karsten: In our case, not so much, its typically one user running the queries<br><br />
Karsten: In our environments, queries can easily run for 30, 60 minutes, or longer<br />
<br />
Michael Paquier: There is a memory debug function in Postgres (the same that prints in OOM situations), that could be utilized<br><br />
Joe Conway: Someone needs to sit down and come up with a list of requirements on resource management<br />
<br />
Michael Paquier: One idea could be to have a per-session memory limit that is managed by each backend<br><br />
Joe Conway: Whatever you do in this area, it would have to not be overrideable per-user (from a security perspective)<br />
<br />
[ discussion around measuring whether parallel query works ]<br />
<br />
(?): There are two counters in shared memory around parallel workers launched/etc that could be exported<br />
<br />
<br />
=== Hypothetical Partitions for testing optimizations ===<br />
<br />
- like hypopg does for indexes, but with partitions<br />
<br />
challenges:<br />
- how to get a fake table name, and how this is already accomplished for index relations in hypopg<br />
- relcache does not have the ability to remove negative relcache entries<br />
- how to impose catalog entries to represent the partitions in just one session<br />
- velocity of changes to partitioning in recent versions (10,11, master) means that few people understand what current behavior is<br />
- some insert/delete hooks coming in v12, which may cut down the amount of code that needs to be duplicated<br />
- CreateFakeRelcacheEntry / FreeFakeRelcacheEntry<br />
- How to select Oids for fake objects on a read replica, risk collision with new real objects from master. Some question of whether fake Oids a<br />
re even necessary.<br />
<br />
It was suggested that refactoring for v12, even if incomplete, is better than no progress at all.<br />
<br />
Several user questions about hypopg and how stable it is, and how it would be used, with an eye toward including in cloud DBAAS products.<br />
<br />
Some concern was expressed that it may be too late in the development cycle to get a chance of this size into v12.<br />
<br />
=== Partition-wise Join & Visibility Map ===<br />
<br />
- KaiGai Kohei<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2016_Developer_Unconference PgConf.ASIA 2016 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2017_Developer_Unconference PgCon 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PGConf.ASIA2017_Developer_Unconference PgConf.ASIA 2017 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2018_Developer_Unconference PgCon 2018 Developer Unconference]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGStrom&diff=31856PGStrom2018-04-17T08:38:03Z<p>Kaigai: </p>
<hr />
<div>'''IMPORTANT NOTICE'''<br />
<br />
Description of this wikipage is not maintained for more than a year.<br />
Please reference the online manual of the PG-Strom project instead.<br />
<br />
* PG-Strom Github: [https://github.com/heterodb/pg-strom https://github.com/heterodb/pg-strom]<br />
* PG-Strom Documentation: [http://heterodb.github.io/pg-strom/ http://heterodb.github.io/pg-strom/]<br />
* PG-Strom Documentation(Japanese): [http://heterodb.github.io/pg-strom/ja/ http://heterodb.github.io/pg-strom/ja/]<br />
<br />
'''NOTE''': Deprecated documentation was removed to avoid confusion (22-Feb-2017)</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2017_Developer_Unconference&diff=31253PGConf.ASIA2017 Developer Unconference2017-12-04T05:20:21Z<p>Kaigai: /* Asian community affairs */</p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 4 Dec, 2017 at Akihabara Conference Center, as part of [http://www.pgconf.asia/EN/2017/ PGConf.ASIA 2017]. <br />
<br />
This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure.<br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Room 1<br />
!Room 2<br />
|- style="background-color:lightgray;"<br />
|9:00-10:00<br />
|<br />
|Registration and theme proposal <br />
<br />
|- style="background-color:lightgray;"<br />
|10:00-10:30<br />
|<br />
|Arrangement & theme decision<br />
<br />
|-<br />
|10:30-11:20 <br />
|GPU Acceleration<br />
|Diagrams and pictures in the docs<br />
<br />
|-<br />
|11:30-12:20 <br />
|SCRAM improvements<br />
|Upgrading without downtime<br />
<br />
|- style="background-color:lightgray;"<br />
|12:20-13:30 <br />
|Lunch<br />
|Lunch<br />
<br />
|-<br />
|13:30-14:20<br />
|Asian community affairs<br />
|Mailing list enhancements<br />
<br />
|-<br />
|14:30-15:20<br />
|NoSQL in Postgres<br />
|Sharding<br />
<br />
|- style="background-color:lightgray;"<br />
|15:50-15:40<br />
|Coffee<br />
|Coffee<br />
<br />
|-<br />
|15:40-16:30<br />
|Logical replication<br />
|Partitioning<br />
<br />
|- style="background-color:lightgray;"<br />
|16:30-17:00<br />
|<br />
|Group photo<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
<br />
=== GPU Acceleration ===<br />
<br />
Coordinator: KaiGai Kohei<br />
<br />
KaiGai introduced the recent updates of GPU acceleration on PostgreSQL; especially, two major features.<br />
The first one is '''SSD-to-GPU Direct SQL Execution''' for I/O heavy workloads. The other one is '''PL/CUDA''' for calculation intensive workloads.<br />
<br />
'''SSD-to-GPU Direct SQL Execution''' utilizes P2P DMA from SSD to GPU, then GPU processes the pre-compiled SQL workloads to reduce amount of the records processed by CPU. It eventually looks like I/O acceleration by cooperation of SSD/GPU.<br />
As a future improvement, he mention about columnar-cache (in v2.0) and intelligent storage concept (v3.0).<br />
The columnar-cache constructs in-memory cache with conversion to the columnar-format. GPU kernel can switch its kernel according to existence of the cache.<br />
<br />
One question from the audience was integration with the '''pluggable storage''' project in PostgreSQL. It has both of advantage and disadvantage. The columnar-store is persistent entity, so it has to involve storage write on updates, on the other hands, no need to convert the storage format if storage layer already has raw data in columnar format. KaiGai is positive to support columnar-storage also, once PostgreSQL core support a columnar storage engine.<br />
<br />
'''Intelligent Storage''' is a concept to utilize special SSD products that allow to mount custom logic on storage side. Once we could implement a logic to convert row-data of PostgreSQL to columnar format prior to send packet over PCIe, it can reduce the PCIe bandwidth consumption and can pull out maximum performance of GPU.<br />
<br />
One other discussion from the audience was enhancement of interfaces. Now PostgreSQL has well designed '''custom-scan interface'''; that allows to generate GPU programs and compile it on the fly, then callbacks on executor allows extension to switch optimal way to scan the source tables. If tablespace is constructed on NVMe-SSD and table-size is enough larger than physical RAM, PG-Strom switches SSD2GPU direct instead of the normal buffered read. Once intelligent storage mechanism got supported, a new alternative option will be added.<br />
<br />
=== Diagrams and pictures in the docs ===<br />
<br />
=== SCRAM improvements ===<br />
<br />
=== Upgrading without downtime ===<br />
<br />
=== Lunch ===<br />
<br />
=== Asian community affairs ===<br />
<br />
Coordinator: Hemmi<br />
<br />
Introduction of local communities: Indonesia, India, South Korea, Philippines, Vietnam, Taiwan, Japan<br />
<br />
Indonesia:<br />
How did it start at Indonesia? Began from training session, not enterprise origin more than 10 years before.<br />
<br />
India:<br />
Very strong community base. PGconf.India expects 2000-3000 participants. EDB pays much efforts to promote PostgreSQL in India.<br />
Some India folks have contributed core features like FDW, partitioning and so on.<br />
<br />
South Korea:<br />
Facebook group in South Korea is one of the most active community, like technical documentation translation.<br />
How open source accepted in enterprise of South Korea? Usually OracleDB, but a startup back adopted MySQL 5 years before.<br />
<br />
Philippines:<br />
Little active community, however, some companies offer training services in enterprise class.<br />
SMB or academic mainly uses open source for cost reduction. <br />
(Here is a case of PostgreSQL at small telecommunications and banking customers.)<br />
<br />
Vietnam:<br />
Academic user is majority of user base. (presenter's point of view)<br />
User community was launched at Jul-2017, with 10 members. The first meetup was held.<br />
They want to provide .po files for Vietnam language.<br />
<br />
Taiwan:<br />
<br />
Japan:<br />
Two major community for PostgreSQL. JPUG; grass roots users group more than 15years. PGEcons; a consortium supported by enterprises.<br />
Which is majority users? Academic/Education or Enterprise? Likely, enterprise users is majority.<br />
Japan has large user base. (Likely, Russian user has grown rapidly.)<br />
What is a good idea to promote PostgreSQL in other nations also.<br />
Education and certification of professional are significant; OSS-DB program by LPI Japan.<br />
No certification cannot cover all type of the skill-set.<br />
<br />
=== Mailing list enhancements ===<br />
<br />
=== NoSQL in Postgres ===<br />
<br />
=== Sharding ===<br />
<br />
=== Logical replication ===<br />
<br />
=== Partitioning ===<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
* TBD<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [[PGConf.ASIA2016_Developer_Unconference|PgConf.ASIA 2016 Developer Unconference]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2017_Developer_Unconference&diff=31252PGConf.ASIA2017 Developer Unconference2017-12-04T03:06:17Z<p>Kaigai: /* GPU Acceleration */</p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 4 Dec, 2017 at Akihabara Conference Center, as part of [http://www.pgconf.asia/EN/2017/ PGConf.ASIA 2017]. <br />
<br />
This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure.<br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Room 1<br />
!Room 2<br />
|- style="background-color:lightgray;"<br />
|9:00-10:00<br />
|<br />
|Registration and theme proposal <br />
<br />
|- style="background-color:lightgray;"<br />
|10:00-10:30<br />
|<br />
|Arrangement & theme decision<br />
<br />
|-<br />
|10:30-11:20 <br />
|GPU Acceleration<br />
|Diagrams and pictures in the docs<br />
<br />
|-<br />
|11:30-12:20 <br />
|SCRAM improvements<br />
|Upgrading without downtime<br />
<br />
|- style="background-color:lightgray;"<br />
|12:20-13:30 <br />
|Lunch<br />
|Lunch<br />
<br />
|-<br />
|13:30-14:20<br />
|Asian community affairs<br />
|Mailing list enhancements<br />
<br />
|-<br />
|14:30-15:20<br />
|NoSQL in Postgres<br />
|Sharding<br />
<br />
|- style="background-color:lightgray;"<br />
|15:50-15:40<br />
|Coffee<br />
|Coffee<br />
<br />
|-<br />
|15:40-16:30<br />
|Logical replication<br />
|Partitioning<br />
<br />
|- style="background-color:lightgray;"<br />
|16:30-17:00<br />
|<br />
|Group photo<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
<br />
=== GPU Acceleration ===<br />
<br />
Coordinator: KaiGai Kohei<br />
<br />
KaiGai introduced the recent updates of GPU acceleration on PostgreSQL; especially, two major features.<br />
The first one is '''SSD-to-GPU Direct SQL Execution''' for I/O heavy workloads. The other one is '''PL/CUDA''' for calculation intensive workloads.<br />
<br />
'''SSD-to-GPU Direct SQL Execution''' utilizes P2P DMA from SSD to GPU, then GPU processes the pre-compiled SQL workloads to reduce amount of the records processed by CPU. It eventually looks like I/O acceleration by cooperation of SSD/GPU.<br />
As a future improvement, he mention about columnar-cache (in v2.0) and intelligent storage concept (v3.0).<br />
The columnar-cache constructs in-memory cache with conversion to the columnar-format. GPU kernel can switch its kernel according to existence of the cache.<br />
<br />
One question from the audience was integration with the '''pluggable storage''' project in PostgreSQL. It has both of advantage and disadvantage. The columnar-store is persistent entity, so it has to involve storage write on updates, on the other hands, no need to convert the storage format if storage layer already has raw data in columnar format. KaiGai is positive to support columnar-storage also, once PostgreSQL core support a columnar storage engine.<br />
<br />
'''Intelligent Storage''' is a concept to utilize special SSD products that allow to mount custom logic on storage side. Once we could implement a logic to convert row-data of PostgreSQL to columnar format prior to send packet over PCIe, it can reduce the PCIe bandwidth consumption and can pull out maximum performance of GPU.<br />
<br />
One other discussion from the audience was enhancement of interfaces. Now PostgreSQL has well designed '''custom-scan interface'''; that allows to generate GPU programs and compile it on the fly, then callbacks on executor allows extension to switch optimal way to scan the source tables. If tablespace is constructed on NVMe-SSD and table-size is enough larger than physical RAM, PG-Strom switches SSD2GPU direct instead of the normal buffered read. Once intelligent storage mechanism got supported, a new alternative option will be added.<br />
<br />
=== Diagrams and pictures in the docs ===<br />
<br />
=== SCRAM improvements ===<br />
<br />
=== Upgrading without downtime ===<br />
<br />
=== Lunch ===<br />
<br />
=== Asian community affairs ===<br />
<br />
=== Mailing list enhancements ===<br />
<br />
=== NoSQL in Postgres ===<br />
<br />
=== Sharding ===<br />
<br />
=== Logical replication ===<br />
<br />
=== Partitioning ===<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
* TBD<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [[PGConf.ASIA2016_Developer_Unconference|PgConf.ASIA 2016 Developer Unconference]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2017_Developer_Unconference&diff=31251PGConf.ASIA2017 Developer Unconference2017-12-04T03:05:39Z<p>Kaigai: /* GPU Acceleration */</p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 4 Dec, 2017 at Akihabara Conference Center, as part of [http://www.pgconf.asia/EN/2017/ PGConf.ASIA 2017]. <br />
<br />
This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure.<br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Room 1<br />
!Room 2<br />
|- style="background-color:lightgray;"<br />
|9:00-10:00<br />
|<br />
|Registration and theme proposal <br />
<br />
|- style="background-color:lightgray;"<br />
|10:00-10:30<br />
|<br />
|Arrangement & theme decision<br />
<br />
|-<br />
|10:30-11:20 <br />
|GPU Acceleration<br />
|Diagrams and pictures in the docs<br />
<br />
|-<br />
|11:30-12:20 <br />
|SCRAM improvements<br />
|Upgrading without downtime<br />
<br />
|- style="background-color:lightgray;"<br />
|12:20-13:30 <br />
|Lunch<br />
|Lunch<br />
<br />
|-<br />
|13:30-14:20<br />
|Asian community affairs<br />
|Mailing list enhancements<br />
<br />
|-<br />
|14:30-15:20<br />
|NoSQL in Postgres<br />
|Sharding<br />
<br />
|- style="background-color:lightgray;"<br />
|15:50-15:40<br />
|Coffee<br />
|Coffee<br />
<br />
|-<br />
|15:40-16:30<br />
|Logical replication<br />
|Partitioning<br />
<br />
|- style="background-color:lightgray;"<br />
|16:30-17:00<br />
|<br />
|Group photo<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
<br />
=== GPU Acceleration ===<br />
<br />
Coordinator: KaiGai Kohei<br />
<br />
KaiGai introduced the recent updates of GPU acceleration on PostgreSQL; especially, two major features.<br />
The first one is '''SSD-to-GPU Direct SQL Execution''' for I/O heavy workloads. The other one is '''PL/CUDA''' for calculation intensive workloads.<br />
<br />
'''SSD-to-GPU Direct SQL Execution''' utilizes P2P DMA from SSD to GPU, then GPU processes the pre-compiled SQL workloads to reduce amount of the records processed by CPU. It eventually looks like I/O acceleration by cooperation of SSD/GPU.<br />
As a future improvement, he mention about columnar-cache (in v2.0) and intelligent storage concept (v3.0).<br />
The columnar-cache constructs in-memory cache with conversion to the columnar-format. GPU kernel can switch its kernel according to existence of the cache.<br />
<br />
One question from the audience was integration with the '''pluggable storage''' project in PostgreSQL. It has both of advantage and disadvantage. The columnar-store is persistent entity, so it has to involve storage write on updates, on the other hands, no need to convert the storage format if storage layer already has raw data in columnar format. KaiGai is positive to support columnar-storage also, once PostgreSQL core support a columnar storage engine.<br />
<br />
'''Intelligent Storage''' is a concept to utilize special SSD products that allow to mount custom logic on storage side. Once we could implement a logic to convert row-data of PostgreSQL to columnar format prior to send packet over PCIe, it can reduce the PCIe bandwidth consumption and can pull out maximum performance of GPU.<br />
<br />
One other discussion from the audience was enhancement of interfaces. Now PostgreSQL has well designed custom-scan interface; that allows to generate GPU programs and compile it on the fly, then callbacks on executor allows extension to switch optimal way to scan the source tables. If tablespace is constructed on NVMe-SSD and table-size is enough larger than physical RAM, PG-Strom switches SSD2GPU direct instead of the normal buffered read. Once intelligent storage mechanism got supported, a new alternative option will be added.<br />
<br />
=== Diagrams and pictures in the docs ===<br />
<br />
=== SCRAM improvements ===<br />
<br />
=== Upgrading without downtime ===<br />
<br />
=== Lunch ===<br />
<br />
=== Asian community affairs ===<br />
<br />
=== Mailing list enhancements ===<br />
<br />
=== NoSQL in Postgres ===<br />
<br />
=== Sharding ===<br />
<br />
=== Logical replication ===<br />
<br />
=== Partitioning ===<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
* TBD<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [[PGConf.ASIA2016_Developer_Unconference|PgConf.ASIA 2016 Developer Unconference]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2017_Developer_Unconference&diff=31250PGConf.ASIA2017 Developer Unconference2017-12-04T02:31:31Z<p>Kaigai: /* Unconference minutes */</p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held on 4 Dec, 2017 at Akihabara Conference Center, as part of [http://www.pgconf.asia/EN/2017/ PGConf.ASIA 2017]. <br />
<br />
This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure.<br />
<br />
== Unconference Tokyo Round Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Room 1<br />
!Room 2<br />
|- style="background-color:lightgray;"<br />
|9:00-10:00<br />
|<br />
|Registration and theme proposal <br />
<br />
|- style="background-color:lightgray;"<br />
|10:00-10:30<br />
|<br />
|Arrangement & theme decision<br />
<br />
|-<br />
|10:30-11:20 <br />
|GPU Acceleration<br />
|Diagrams and pictures in the docs<br />
<br />
|-<br />
|11:30-12:20 <br />
|SCRAM improvements<br />
|Upgrading without downtime<br />
<br />
|- style="background-color:lightgray;"<br />
|12:20-13:30 <br />
|Lunch<br />
|Lunch<br />
<br />
|-<br />
|13:30-14:20<br />
|Asian community affairs<br />
|Mailing list enhancements<br />
<br />
|-<br />
|14:30-15:20<br />
|NoSQL in Postgres<br />
|Sharding<br />
<br />
|- style="background-color:lightgray;"<br />
|15:50-15:40<br />
|Coffee<br />
|Coffee<br />
<br />
|-<br />
|15:40-16:30<br />
|Logical replication<br />
|Partitioning<br />
<br />
|- style="background-color:lightgray;"<br />
|16:30-17:00<br />
|<br />
|Group photo<br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
<br />
=== GPU Acceleration ===<br />
<br />
=== Diagrams and pictures in the docs ===<br />
<br />
=== SCRAM improvements ===<br />
<br />
=== Upgrading without downtime ===<br />
<br />
=== Lunch ===<br />
<br />
=== Asian community affairs ===<br />
<br />
=== Mailing list enhancements ===<br />
<br />
=== NoSQL in Postgres ===<br />
<br />
=== Sharding ===<br />
<br />
=== Logical replication ===<br />
<br />
=== Partitioning ===<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Venue<br />
* TBD<br />
<br />
== Notes ==<br />
* TBD<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]<br />
** [[PGConf.ASIA2016_Developer_Unconference|PgConf.ASIA 2016 Developer Unconference]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=Foreign_data_wrappers&diff=31154Foreign data wrappers2017-11-12T08:33:00Z<p>Kaigai: </p>
<hr />
<div>= Foreign Data Wrappers =<br />
In 2003, a new specification called [[SQL/MED]] ("SQL Management of External Data") was added to the SQL standard. It is a standardized way of handling access to remote objects from SQL databases. In 2011, PostgreSQL 9.1 was released with read-only support of this standard, and in 2013 write support was added with PostgreSQL 9.3.<br />
<br />
There are now a variety of Foreign Data Wrappers (FDW) available which enable PostgreSQL Server to different remote data stores, ranging from other SQL databases through to flat file. This page list some of the wrappers currently available. Another [https://pgxn.org/tag/fdw/ fdw list] can be found at [https://pgxn.org/ the PGXN website].<br />
<br />
Please keep in mind that most of these wrappers are '''not officially supported by the PostgreSQL Global Development Group''' (PGDG) and that some of these projects are '''still in Beta''' version. Use carefully!<br />
<br />
<br />
== Generic SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/ZhengYang/odbc_fdw github]<br />
|[https://pgxn.org/dist/odbc_fdw/ PGXN]<br />
|[http://www.postgresonline.com/journal/archives/249-ODBC-Foreign-Data-wrapper-to-query-SQL-Server-on-Window---Part-2.html example] <br />
|Does not compile with PostgreSQL >= 9.2!<br />
|-<br />
|ODBC<br />
|Native<br />
|<br />
|[https://github.com/CartoDB/odbc_fdw github]<br />
|<br />
|<br />
|CartoDB took over active development of the ODBC FDW for PG 9.5+ <br />
|-<br />
|JDBC<br />
|Native<br />
|<br />
|[https://github.com/atris/JDBC_FDW github]<br />
|<br />
|<br />
| Not maintained ?<br />
|-<br />
|JDBC2<br />
|Native<br />
|<br />
|[https://github.com/heimir-sverrisson/jdbc2_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
| [http://www.sqlalchemy.org/ SQL_Alchemy]<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN]<br />
| [http://multicorn.org/foreign-data-wrappers/#sqlalchemy-foreign-data-wrapper documentation]<br />
| Can be used to access data stored in any database supported by the sqlalchemy python toolkit.<br />
|-<br />
| VirtDB<br />
| Native<br />
| GPL<br />
| [https://github.com/virtdb/virtdb-fdw GitHub]<br />
|<br />
|<br />
| A generic FDW to access VirtDB data sources (SAP ERP, Oracle RDBMS)<br />
|}<br />
<br />
== Specific SQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://www.postgresql.org/ PostgreSQL]<br />
|Native<br />
|PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
|[https://www.postgresql.org/docs/current/static/postgres-fdw.html documentation]<br />
| <br />
|-<br />
|[https://www.oracle.com/index.html Oracle]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/laurenz/oracle_fdw github]<br />
|[https://pgxn.org/dist/oracle_fdw/ PGXN]<br />
|[http://laurenz.github.io/oracle_fdw/ website]<br />
| <br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/postgresql-91-meet-mysql example]<br />
|An early version of the Foreign Data Wrapper for MySQL that supports PostgreSQL 9.1 and above: <br />
|-<br />
|[https://www.mysql.com/ MySQL]<br />
|Native<br />
|<br />
|[https://github.com/EnterpriseDB/mysql_fdw github]<br />
|[https://pgxn.org/dist/mysql_fdw/ PGXN]<br />
|[https://www.enterprisedb.com/blog/new-oss-tool-links-postgres-and-mysql example]<br />
|An updated FDW for MySQL that support PostgreSQL 9.3 and above <br />
|-<br />
|Informix<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/credativ/informix_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|Firebird<br />
|Native<br />
|<br />
|[https://github.com/ibarwick/firebird_fdw/ github]<br />
|[https://pgxn.org/dist/firebird_fdw/ PGXN]<br />
|<br />
| currently work-in-progress.<br />
|-<br />
|SQLite<br />
|Native<br />
|<br />
|[https://github.com/gleu/sqlite_fdw github]<br />
|<br />
|<br />
|An FDW for SQLite3 (read-only)<br />
|-<br />
|Sybase / MS SQL Server<br />
|Native<br />
|<br />
|[https://github.com/tds-fdw/tds_fdw github]<br />
|[https://pgxn.org/dist/tds_fdw/ PGXN]<br />
|<br />
|An FDW for Sybase and Microsoft SQL server<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== NoSQL Database Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[https://cloud.google.com/bigtable/ BigTable or HBase]<br />
|[https://github.com/posix4e/rpgffi Native Rust Binding (RPGFFI)]<br />
|MIT<br />
|[https://github.com/durch/google-bigtable-postgres-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://cassandra.apache.org/ Cassandra]<br />
|[http://multicorn.org/ Multicorn]<br />
|MIT<br />
|[https://github.com/rankactive/cassandra-fdw Github]<br />
|[https://rankactive.com/resources/postgresql-cassandra-fdw Rankactive]<br />
|<br />
|<br />
|-<br />
| Cassandra2<br />
| Native<br />
| MIT<br />
|[https://github.com/jaiminpan/cassandra2_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|-<br />
| [http://cassandra.apache.org Cassandra]<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/wjch-krl/pgCassandra Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://clickhouse.yandex/ ClickHouse]<br />
|[http://multicorn.org/ Multicorn]<br />
|BSD<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/ Github]<br />
|<br />
|[https://github.com/Infinidat/infi.clickhouse_fdw/blob/master/README.md README]<br />
|<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/ZhengYang/couchdb_fdw Github]<br />
|[https://pgxn.org/dist/couchdb_fdw/ PGXN]<br />
|<br />
| Original version<br />
|-<br />
|[http://couchdb.apache.org/ CouchDB]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/golgauth/couchdb_fdw Github]<br />
|<br />
|<br />
| golgauth version (9.1 - 9.2+ compatible)<br />
|-<br />
|[http://fallabs.com/kyototycoon/ Kyoto Tycoon ]<br />
|Native<br />
|MIT<br />
|[https://github.com/cloudflare/kt_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
|Native<br />
|GPL3+<br />
|[https://github.com/EnterpriseDB/mongo_fdw Github]<br />
|[https://pgxn.org/dist/mongo_fdw/ PGXN]<br />
|[https://github.com/EnterpriseDB/mongo_fdw/blob/master/README.md README]<br />
|EDB version<br />
|-<br />
|[https://www.mongodb.com/ MongoDB]<br />
| [http://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/dwa/mongoose_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://www.mongodb.com/ MongoDB]<br />
| [http://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/asya999/yam_fdw Github]<br />
|<br />
|<br />
| Yet Another Postgres FDW for MongoDB<br />
|-<br />
|[https://neo4j.com/ Neo4j]<br />
|Native<br />
|?<br />
|[https://github.com/nuko-yokohama/neo4j_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://quasar-analytics.org/ Quasar]<br />
|Native<br />
|Apache<br />
|[https://github.com/quasar-analytics/quasar-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[https://redis.io/ Redis]<br />
|Native<br />
|PostgreSQL<br />
|[https://github.com/pg-redis-fdw/redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://redis.io/ Redis]<br />
| Native<br />
| BSD<br />
| [https://github.com/nahanni/rw_redis_fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
| [https://rethinkdb.com/ RethinkDB]<br />
| [http://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/rotten/rethinkdb-multicorn-postgresql-fdw Github]<br />
|<br />
| [https://rethinkdb.com/blog/postgres-foreign-data-wrapper/ blog]<br />
|<br />
|-<br />
| [https://github.com/basho/riak Riak]<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/kiskovacs/riak-multicorn-pg-fdw Github]<br />
|<br />
|<br />
|<br />
|-<br />
|[http://whitedb.org/ WhiteDB]<br />
| Native<br />
| MIT<br />
| [https://github.com/Kentik/wdb_fdw Github]<br />
|<br />
|<br />
|<br />
|}<br />
<br />
== File Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| CSV<br />
| Native<br />
| PostgreSQL<br />
|[https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/file_fdw;hb=HEAD git.postgresql.org]<br />
|<br />
| [https://www.postgresql.org/docs/current/static/file-fdw.html documentation]<br />
| Delivered as an official extension of PostgreSQL 9.1 / [https://www.depesz.com/2011/03/14/waiting-for-9-1-foreign-data-wrapper/ example] / [http://www.postgresonline.com/journal/archives/250-File-FDW-Family-Part-1-file_fdw.html Another example]<br />
|-<br />
| CSV<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| [http://multicorn.org/foreign-data-wrappers/#csv-foreign-data-wrapper documentation]<br />
| Each column defined in the table will be mapped, in order, against columns in the CSV file.<br />
|-<br />
| CSV / Text Array<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_text_array_fdw GitHub]<br />
|<br />
| [http://www.postgresonline.com/journal/archives/251-File-FDW-Family-Part-2-file_textarray_fdw-Foreign-Data-Wrapper.html How to] <br />
| Another CSV wrapper<br />
|-<br />
| CSV / Fixed-length<br />
| Native<br />
|<br />
| [https://github.com/adunstan/file_fixed_length_record_fdw GitHub]<br />
|<br />
| <br />
| <br />
|-<br />
| CSV / gzipped<br />
| [http://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/dialogbox/py_csvgz_fdw GitHub]<br />
|<br />
| <br />
| PostgreSQL Foreign Data Wrapper for gzipped cvs file<br />
|-<br />
| Compressed File<br />
| Native<br />
|<br />
| [https://github.com/gokhankici/compressedfile_fdw GitHub]<br />
|<br />
| <br />
| <br />
|-<br />
| Document Collection<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/ZhengYang/dc_fdw GitHub]<br />
|<br />
| [https://github.com/ZhengYang/dc_fdw/wiki wiki]<br />
| <br />
|-<br />
| JSON<br />
| Native<br />
| GPL3<br />
| [https://github.com/nkhorman/json_fdw GitHub]<br />
|<br />
| [https://www.citusdata.com/blog/2013/05/30/run-sql-on-json-files-without-any-data-loads/ Example]<br />
| <br />
|-<br />
| Multi-File<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| [http://multicorn.org/foreign-data-wrappers/#filesystem-foreign-data-wrapper doc]<br />
| Access data stored in various files in a filesystem. The files are looked up based on a pattern, and parts of the file's path are mapped to various columns, as well as the file's content itself.<br />
|-<br />
| Multi CDR<br />
| Native<br />
| PostgreSQL <br />
| [https://github.com/theirix/multicdr_fdw GitHub]<br />
| [https://pgxn.org/dist/multicdr_fdw/ PGXN] <br />
| <br />
| <br />
|-<br />
| pg_dump<br />
| Native<br />
| New BSD<br />
| [https://github.com/MeetMe/dump_fdw GitHub]<br />
|<br />
|<br />
| Allows querying of data directly against Postgres custom format files created by pg_dump<br />
|-<br />
| TAR Files<br />
| Native<br />
| <br />
| [https://github.com/beargiles/tarfile-fdw GitHub]<br />
|<br />
|<br />
| <br />
|-<br />
| XML<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| <br />
|<br />
|-<br />
| ZIP Files<br />
| Native<br />
| <br />
| [https://github.com/beargiles/zipfile-fdw GitHub]<br />
|<br />
|<br />
| <br />
|}<br />
<br />
== Geo Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|[http://www.gdal.org GDAL/OGR]<br />
|Native<br />
|MIT<br />
|[https://github.com/pramsey/pgsql-ogr-fdw GitHub]<br />
|<br />
|<br />
|A wrapper for data sources with a [http://www.gdal.org GDAL/OGR] driver, including databases like Oracle, Informix, SQLite, SQL Server, ODBC as well as file formats like Shape, FGDB, MapInfo, CSV, Excel, OpenOffice, OpenStreetMap PBF and XML, OGC WebServices, [http://www.gdal.org/ogr_formats.html and more] Spatial columns are linked in as PostGIS geometry if PostGIS is installed.<br />
|-<br />
| Geocode / GeoJSON <br />
| [http://multicorn.org/ Multicorn]<br />
| GPL<br />
| [https://github.com/bosth/geofdw GitHub]<br />
|<br />
|<br />
| a collection of PostGIS-related foreign data wrappers<br />
|-<br />
| [http://wiki.openstreetmap.org/wiki/PBF_Format Open Street Map PBF] <br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/vpikulik/postgres_osm_pbf_fdw GitHub]<br />
|<br />
|<br />
| <br />
|}<br />
<br />
== LDAP Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| LDAP<br />
| Native<br />
| <br />
| [https://github.com/guedes/ldap_fdw GitHub]<br />
| [https://pgxn.org/dist/ldap_fdw/ PGXN]<br />
| <br />
| Allows to query an LDAP server and retrieve data from some pre-configured Organizational Unit<br />
|-<br />
| LDAP<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| [http://multicorn.org/foreign-data-wrappers/#idldap-foreign-data-wrapper documentation]<br />
| <br />
|}<br />
<br />
<br />
== Generic Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Git<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| <br />
| <br />
|-<br />
| Git<br />
| Native<br />
| MIT<br />
| [https://github.com/franckverrot/git_fdw GitHub]<br />
|<br />
| <br />
| <br />
|-<br />
| ICAL<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/daamien/Multicorn/blob/master/python/multicorn/icalfdw.py GitHub]<br />
| <br />
| [https://wiki.postgresql.org/images/7/7e/Conferences-write_a_foreign_data_wrapper_in_15_minutes-presentation.pdf pdf]<br />
| <br />
|-<br />
| IMAP<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| [http://multicorn.org/foreign-data-wrappers/#idimap-foreign-data-wrapper documentation]<br />
| <br />
|-<br />
| RSS<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| [http://multicorn.org/foreign-data-wrappers/#idrss-foreign-data-wrapper documentation]<br />
| This fdw can be used to access items from an rss feed. <br />
|-<br />
| www<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/cyga/www_fdw/ GitHub]<br />
| [https://pgxn.org/dist/www_fdw/ PGXN]<br />
| [https://github.com/cyga/www_fdw/wiki wiki]<br />
| Allows to query different web services<br />
|}<br />
<br />
== Specific Web Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Database.com<br />
| [http://multicorn.org/ Multicorn]<br />
| BSD <br />
| [https://github.com/metadaddy/Database.com-FDW-for-PostgreSQL GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Dun & Badstreet<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/dpmorel/dnb_fdw GitHub]<br />
| <br />
| <br />
| Access to the [https://fr.wikipedia.org/wiki/Data_Universal_Numbering_System Data Universal Numbering System] (DUNS)<br />
|-<br />
| DynamoDB<br />
| [http://multicorn.org/ Multicorn]<br />
| GPL <br />
| [https://github.com/avances123/dynamodb_fdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Facebook<br />
| [http://multicorn.org/ Multicorn]<br />
| <br />
| [https://github.com/mrwilson/fb-psql GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Fixer.io<br />
| based on www_fdw<br />
| <br />
| [https://github.com/hakanensari/fixer-io GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Google<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/Kozea/Multicorn GitHub]<br />
| [https://pgxn.org/dist/multicorn/ PGXN] <br />
| <br />
| <br />
|-<br />
| Heroku dataclips<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/petergeoghegan/dataclips_fdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Mailchimp<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL <br />
| [https://github.com/daamien/mailchimp_fdw GitHub]<br />
| <br />
| <br />
| Beta <br />
|-<br />
| [http://parseplatform.org/ Parse]<br />
| [http://multicorn.org/ Multicorn]<br />
| MIT <br />
| [https://github.com/spacialdb/parse_fdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| S3<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/s3_fdw GitHub]<br />
| [https://pgxn.org/dist/s3_fdw/ PGXN]<br />
| <br />
| <br />
|-<br />
| S3CSV<br />
| [http://multicorn.org/ Multicorn]<br />
| GPL 3<br />
| [https://github.com/eligoenergy/s3csv_fdw GitHub]<br />
| <br />
| <br />
| This is meant to replace s3_fdw that does is not supported on PostgreSQL version 9.2+<br />
|-<br />
| Twitter<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/umitanuki/twitter_fdw GitHub]<br />
| [https://pgxn.org/dist/twitter_fdw/ PGXN]<br />
| <br />
| A wrapper fetching text messages from Twitter over the Internet and returning a table<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| Native<br />
| Apache<br />
| [https://github.com/komamitsu/treasuredata_fdw GitHub]<br />
| [https://pgxn.org/dist/treasuredata_fdw PGXN]<br />
| <br />
| A FDW for Treasure Data internally using a Rust library<br />
|-<br />
| [https://www.treasuredata.com/ Treasure Data]<br />
| [http://multicorn.org/ Multicorn]<br />
| Apache<br />
| [https://github.com/komamitsu/td-fdw GitHub]<br />
|<br />
| <br />
|<br />
|-<br />
| Google Spreadsheets<br />
| [http://multicorn.org/ Multicorn]<br />
| MIT<br />
| [https://github.com/lincolnturner/gspreadsheet_fdw GitHub]<br />
| <br />
| <br />
| <br />
|}<br />
<br />
== Big Data Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Elastic Search<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
|[https://github.com/Mikulas/pg-es-fdw GitHub]<br />
|<br />
|<br />
|<br />
|-<br />
| Google BigQuery<br />
| [http://multicorn.org/ Multicorn]<br />
| MIT<br />
|[https://github.com/gabfl/bigquery_fdw GitHub]<br />
|<br />
|[https://github.com/gabfl/bigquery_fdw/blob/master/docs/README.md Documentation]<br />
|bigquery_fdw is a BigQuery FDW compatible with PostgreSQL >= 9.5<br />
|-<br />
| file_fdw-gds (Hadoop)<br />
| Native<br />
| <br />
| [https://github.com/wat4dog/pg-file-fdw-gds GitHub]<br />
| <br />
| <br />
| Hadoop file_fdw is a slightly modified version of PostgreSQL 9.3's file_fdw module.<br />
|-<br />
| Hadoop<br />
| Native<br />
| PostgreSQL<br />
| [https://bitbucket.org/openscg/hadoop_fdw Bitbucket]<br />
| <br />
| <br />
| Allows read and write access to HBase as well as to HDFS via Hive.<br />
|-<br />
| HDFS<br />
| Native<br />
| Apache<br />
| [https://github.com/EnterpriseDB/hdfs_fdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Hive<br />
| [http://multicorn.org/ Multicorn]<br />
| <br />
| [https://github.com/youngwookim/hive-fdw-for-postgresql GitHub]<br />
| <br />
| <br />
| Used to access Apache Hive tables.<br />
|-<br />
| Hive / ORC File<br />
| Native<br />
| <br />
| [https://github.com/gokhankici/orc_fdw GitHub]<br />
| <br />
| <br />
|<br />
|-<br />
| [http://impala.io/ Impala]<br />
| Native<br />
| BSD <br />
| [https://github.com/lapug/impala_fdw GitHub]<br />
| <br />
| <br />
| <br />
|}<br />
<br />
== Column-Oriented Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
|Columnar Store<br />
|Native<br />
|<br />
|[https://github.com/citusdata/cstore_fdw github]<br />
|[https://www.citusdata.com/blog/2014/04/03/columnar-store-for-analytics/ example]<br />
|<br />
|A Columnar Store for PostgreSQL.<br />
|-<br />
|[https://www.monetdb.org/ MonetDB]<br />
|Native<br />
|<br />
|[https://github.com/snaga/monetdb_fdw github]<br />
|<br />
|<br />
|<br />
|-<br />
|GPU Memory Store<br />
|Native<br />
|GPL v2<br />
|[https://github.com/heterodb/pg-strom github]<br />
|<br />
|<br />
|FDW to GPU device memory; a part of PG-Strom feature for PL/CUDA<br />
|}<br />
<br />
== Scientific Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Ambry <br />
| [http://multicorn.org/ Multicorn]<br />
|<br />
| [https://github.com/nmb10/ambryfdw GitHub]<br />
|<br />
| <br />
|<br />
|-<br />
| ROOT files<br />
| Native<br />
|<br />
| [https://github.com/miguel-branco/root_fdw GitHub]<br />
|<br />
| <br />
| https://root.cern.ch<br />
|-<br />
| VCF files (Genotype)<br />
| [http://multicorn.org/ Multicorn]<br />
| <br />
| [https://github.com/smithijk/vcf_fdw_postgresql GitHub]<br />
| <br />
| <br />
| https://en.wikipedia.org/wiki/Variant_Call_Format<br />
|}<br />
<br />
== Operating System Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Docker<br />
| [http://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/dockerfdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Log files<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/rdunklau/logfdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| OpenStack / Telemetry <br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/CSCfi/telemetry-fdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| OS Query<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/shish/pgosquery GitHub]<br />
| <br />
| <br />
| Like Facebook's OSQuery, but for Postgres<br />
|-<br />
| Passwd<br />
| Native<br />
| PostgreSQL<br />
| [https://github.com/beargiles/passwd-fdw GitHub]<br />
| <br />
| <br />
| reads linux/unix password and group files.<br />
|-<br />
| Process<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/Kozea/Multicorn/blob/master/python/multicorn/processfdw.py GitHub]<br />
| <br />
| <br />
| A foreign datawrapper for querying system stats based on [https://www.i-scream.org/libstatgrab/ statgrab]<br />
|}<br />
<br />
== Exotic Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| fdw_fdw<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/daamien/fdw_fdw GitHub]<br />
| <br />
| <br />
| the Meta FDW ! reads this page and returns the list of all the FDW<br />
|-<br />
| PGStrom<br />
| Native<br />
| GPL 2<br />
| [https://github.com/heterodb/pg-strom GitHub]<br />
| <br />
| [http://wiki.postgresql.org/wiki/PGStrom wiki]<br />
| uses GPU devices to accelarate sequential scan on massive amount of records with complex qualifiers. Now it moved to CustomScan based implementation, so its core logic no longer uses FDW.<br />
|-<br />
| PPG<br />
| Native<br />
| <br />
| [https://github.com/scarbrofair/ppg_fdw GitHub]<br />
| <br />
|<br />
| distributed parallel query engine, based on fdw and hooks of PG<br />
|-<br />
| Open Civic Data<br />
| [http://multicorn.org/ Multicorn]<br />
| Expat<br />
| [https://github.com/paultag/sunlightfdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| [http://www2.meethue.com/en-us/about-hue Phillips Hue Lighting Systems]<br />
| [http://multicorn.org/ Multicorn]<br />
| MIT <br />
| [https://github.com/rotten/hue-multicorn-postgresql-fdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Random Number<br />
| [http://multicorn.org/ Multicorn]<br />
| PostgreSQL<br />
| [https://github.com/yieldsfalsehood/rng_fdw GitHub]<br />
| <br />
| <br />
| A random number generator foreign data wrapper for postgres<br />
|-<br />
| Rotfang<br />
| [http://multicorn.org/ Multicorn]<br />
| Native <br />
| [https://bitbucket.org/adunstan/rotfang-fdw BitBucket]<br />
| PostgreSQL<br />
| [https://drive.google.com/file/d/0B3XVAFFWEFN0aURac0dzSFQyZzA/view slides]<br />
| Advanced random number generator <br />
|-<br />
| Template Tables<br />
| Native<br />
| BSD<br />
| [https://github.com/okbob/template_fdw GitHub]<br />
| <br />
| <br />
| PostgreSQL data wrapper for template tables - any DML and SELECT operations are disallowed<br />
|}<br />
<br />
== Example Wrappers ==<br />
<br />
{| align="center" border="1" cellspacing="0" {{Prettytable}}<br />
|-<br />
!{{Hl2}} |Data Source<br />
!{{Hl2}} |Type<br />
!{{Hl2}} |Licence<br />
!{{Hl2}} |Code<br />
!{{Hl2}} |Install<br />
!{{Hl2}} |Doc<br />
!{{Hl2}} |Notes<br />
|-<br />
| Dummy<br />
| Native<br />
| BSD<br />
| [https://github.com/slaught/dummy_fdw GitHub]<br />
| <br />
| <br />
| Readable null FDW for testing<br />
|-<br />
| Hello World<br />
| <br />
| <br />
| [https://github.com/wikrsh/hello_fdw GitHub]<br />
| <br />
| <br />
| <br />
|-<br />
| Black Hole<br />
| <br />
| <br />
| [https://bitbucket.org/adunstan/blackhole_fdw bitbucket]<br />
| <br />
| <br />
| a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
|}<br />
<br />
<br />
=Writing Foreign Database Wrappers=<br />
<br />
* [http://multicorn.org/ Multicorn] is an extension that allows you to write FDWs in Python<br />
* [https://github.com/franckverrot/holycorn Holycorn] is an extension that allows you to write FDWs in Ruby<br />
* [https://www.postgresql.org/docs/current/static/fdwhandler.html Documentation: Writing a Foreign Data Wrapper]<br />
* [https://bitbucket.org/adunstan/blackhole_fdw Black Hole FDW] - a skeleton FDW pre-populated with relevant excerpts from the documentation<br />
* [http://blog.guillaume.lelarge.info/index.php/post/2013/06/25/The-handler-and-the-validator-functions-of-a-FDW FDW tutorial by Guillaume Lelarge]<br />
* [https://github.com/nautilebleu/django-fdw django-fdw] A sample project to test django and Postgres Foreign Data Wrapper<br />
<br />
<br />
[[Category:Foreign-data wrapper|!]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGStrom&diff=29450PGStrom2017-02-22T07:07:00Z<p>Kaigai: Replaced content with "'''IMPORTANT NOTICE'''
Description of this wikipage is not maintained for more than a year.
Please reference the online manual of the PG-Strom project instead.
PG-Strom ..."</p>
<hr />
<div>'''IMPORTANT NOTICE'''<br />
<br />
Description of this wikipage is not maintained for more than a year.<br />
Please reference the online manual of the PG-Strom project instead.<br />
<br />
PG-Strom documentation: [http://strom.kaigai.gr.jp/manual.html http://strom.kaigai.gr.jp/manual.html]<br />
<br />
'''NOTE''': Deprecated documentation was removed to avoid confusion (22-Feb-2017)</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2016_Developer_Unconference&diff=28741PGConf.ASIA2016 Developer Unconference2016-12-01T08:21:09Z<p>Kaigai: </p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held in the afternoon of 1 Dec, 2016 at Akihabara Conference Center, as part of [http://www.pgconf.asia PGConf.ASIA 2016]. <br />
<br />
This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure.<br />
<br />
== Unconference Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Room 1<br />
!Room 2<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 12:00-13:00 <br />
| Theme proposal and registration<br />
| <br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 13:00-13:30 <br />
| Arrangement & theme decision<br />
| <br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 13:30-14:20<br />
| Recovery.conf<br />
| In-memory Columnar Store<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 14:30-15:20<br />
| Commit Fest app Improvement<br />
| Variable length fields over 1GB<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 15:30-16:20<br />
| Pgpool-II & Clustering<br />
| Error code<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 16:30-17:20<br />
| FDW Sharding<br />
| <br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 17:45-<br />
| Social event<br />
| <br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
<br />
=== Recovery.conf ===<br />
<br />
=== In-memory Columnar Store ===<br />
<br />
=== Commit Fest app Improvement ===<br />
<br />
=== Variable length fields over 1GB (KaiGai) ===<br />
<br />
Motivation: we want to give large ( >1GB in this context) data blocks to SQL functions, operators and so on. A typical use case is analytic workloads implemented by procedural language; like PL/CUDA which takes 2D-array as an alternative representation of matrix.<br />
Source of the problem is header format of varlena. The least bit of the first byte introduces the type of header; whether it is short (up to 126B) or long (up to 1GB). The 2nd bit introduces whether the long varlena is compressed or not. Thus, 30bits of 32bits are available to indicate length of the variable length fields. Its maximum length is 1GB.<br />
<br />
xxxxxx00 4-byte length word, aligned, uncompressed data (up to 1G)<br />
xxxxxx10 4-byte length word, aligned, *compressed* data (up to 1G)<br />
00000001 1-byte length word, unaligned, TOAST pointer<br />
xxxxxxx1 1-byte length word, unaligned, uncompressed data (up to 126b)<br />
<br />
In this session, we discussed three ideas to support large variable length fields, especially, the second and third option below.<br />
* Enhancement of varlena header<br />
** Good: Flat data format is available.<br />
** Bad: We cannot expect 4B header is supplied even if it is purely on-memory structure. <br />
* Data type specific solution<br />
** Good: harmless to the existing code.<br />
** Bad: segment boundary around 1GB<br />
** AI: infrastructure enhancement to support type specific format (incl. indirect references) on toast/detoast.<br />
* Utilization of large object<br />
** Good: No infrastructure enhancement is needed.<br />
** Bad: User has to build large objects preliminary, thus, not convenient to construct a large matrix on the fly.<br />
<br />
Through the discussion, overall consensus was type specific solution because most of data types are satisfied with current varlena limitation (<1GB). So, some of data types for specific workloads (like matrix?) will take special support for larger data size. For example, if there is a 8GB matrix we have split into 3x3 chunks, a special matrix data type will be able to have indirect reference to the 9 chunks. And functions/operators which support matrix can support these internal data structure.<br />
A few infrastructure enhancement are expected on toast/detoast routines because toast_save_datum() expects variable length field has its contents in a flat data structure. Likely, type specific callbacks are needed to serialize and deserialize the large flexible length field.<br />
KaiGai will take deeper investigation, then propose the idea to pgsql-hackers. <br />
<br />
=== Pgpool-II & Clustering ===<br />
<br />
=== Error code ===<br />
<br />
=== FDW Sharding ===<br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Please receive 2 drink tickets at the entrance.<br />
* Venue<br />
* PRONTO IL BAR UDX<br />
* https://goo.gl/maps/do5XSrgvRbp<br />
<br />
== Notes ==<br />
* Registration is opened from 12.30pm<br />
* All the attendee will be invited to the social event<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGConf.ASIA2016_Developer_Unconference&diff=28738PGConf.ASIA2016 Developer Unconference2016-12-01T07:44:52Z<p>Kaigai: </p>
<hr />
<div>An Unconference-style event for active PostgreSQL developers will be held in the afternoon of 1 Dec, 2016 at Akihabara Conference Center, as part of [http://www.pgconf.asia PGConf.ASIA 2016]. <br />
<br />
This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure.<br />
<br />
== Unconference Time Table ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Room 1<br />
!Room 2<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 12:00-13:00 <br />
| Theme proposal and registration<br />
| <br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 13:00-13:30 <br />
| Arrangement & theme decision<br />
| <br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 13:30-14:20<br />
| Recovery.conf<br />
| In-memory Columnar Store<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 14:30-15:20<br />
| Commit Fest app Improvement<br />
| Variable length fields over 1GB<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 15:30-16:20<br />
| Pgpool-II & Clustering<br />
| Error code<br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 16:30-17:20<br />
| FDW Sharding<br />
| <br />
<br />
|- style="background-color:lightgray;"<br />
|1 Dec 17:45-<br />
| Social event<br />
| <br />
<br />
|}<br />
<br />
== Unconference minutes ==<br />
<br />
=== Variable length fields over 1GB (KaiGai) ===<br />
<br />
Motivation: we want to give large ( >1GB in this context) data blocks to SQL functions, operators and so on. A typical use case is analytic workloads implemented by procedural language; like PL/CUDA which takes 2D-array as an alternative representation of matrix.<br />
Source of the problem is header format of varlena. The least bit of the first byte introduces the type of header; whether it is short (up to 126B) or long (up to 1GB). The 2nd bit introduces whether the long varlena is compressed or not. Thus, 30bits of 32bits are available to indicate length of the variable length fields. Its maximum length is 1GB.<br />
<br />
xxxxxx00 4-byte length word, aligned, uncompressed data (up to 1G)<br />
xxxxxx10 4-byte length word, aligned, *compressed* data (up to 1G)<br />
00000001 1-byte length word, unaligned, TOAST pointer<br />
xxxxxxx1 1-byte length word, unaligned, uncompressed data (up to 126b)<br />
<br />
In this session, we discussed three ideas to support large variable length fields, especially, the second and third option below.<br />
* Enhancement of varlena header<br />
** Good: Flat data format is available.<br />
** Bad: We cannot expect 4B header is supplied even if it is purely on-memory structure. <br />
* Data type specific solution<br />
** Good: harmless to the existing code.<br />
** Bad: segment boundary around 1GB<br />
** AI: infrastructure enhancement to support type specific format (incl. indirect references) on toast/detoast.<br />
* Utilization of large object<br />
** Good: No infrastructure enhancement is needed.<br />
** Bad: User has to build large objects preliminary, thus, not convenient to construct a large matrix on the fly.<br />
<br />
Through the discussion, overall consensus was type specific solution because most of data types are satisfied with current varlena limitation (<1GB). So, some of data types for specific workloads (like matrix?) will take special support for larger data size. For example, if there is a 8GB matrix we have split into 3x3 chunks, a special matrix data type will be able to have indirect reference to the 9 chunks. And functions/operators which support matrix can support these internal data structure.<br />
A few infrastructure enhancement are expected on toast/detoast routines because toast_save_datum() expects variable length field has its contents in a flat data structure. Likely, type specific callbacks are needed to serialize and deserialize the large flexible length field.<br />
KaiGai will take deeper investigation, then propose the idea to pgsql-hackers. <br />
<br />
== Social Event ==<br />
* All attendees can join the social event.<br />
* Please receive 2 drink tickets at the entrance.<br />
* Venue<br />
* PRONTO IL BAR UDX<br />
* https://goo.gl/maps/do5XSrgvRbp<br />
<br />
== Notes ==<br />
* Registration is opened from 12.30pm<br />
* All the attendee will be invited to the social event<br />
<br />
== Reference ==<br />
* Past unconference events<br />
** [https://wiki.postgresql.org/wiki/Pgcon2013unconference PgCon 2013 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/Pgcon2014unconference PgCon 2014 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Unconference PgCon 2015 Developer Unconference]<br />
** [https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Unconference PgCon 2016 Developer Unconference]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgConf.Asia_2016_Developer_Meeting&diff=28679PgConf.Asia 2016 Developer Meeting2016-11-23T05:04:45Z<p>Kaigai: /* Agenda Items */</p>
<hr />
<div>A meeting of the interested PostgreSQL developers is being planned for the morning of Thursday 1st December, 2016 in Tokyo, prior to PGConf.Asia 2016. In order to keep the numbers manageable, this meeting is by '''invitation only'''. Unfortunately it is quite possible that we've overlooked important individuals during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).<br />
<br />
Please note that the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.6 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies.<br />
<br />
The afternoon will be a Developer Unconference, open to a wider audience.<br />
<br />
This is a PostgreSQL Community event.<br />
<br />
== Meeting Goals ==<br />
<br />
* Review the progress of the 10.0 schedule, and formulate plans to address any issues<br />
* Address any proposed timing, policy, or procedure issues<br />
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br />
<br />
== Time & Location ==<br />
<br />
The event will be held on the fifth floor (using American/Japanese style counting) in room 5A at:<br />
<br />
Akihabara Convention Hall<br />
Akihabara Dai Bldig 4F 1-18-13 Sotokanda, <br />
Chiyoda-ku,<br />
Tokyo 101-0021, <br />
Japan<br />
<br />
Please see the [http://www.akibahall.jp/data/access_eng.html website] for details of how to reach the hall.<br />
<br />
The morning session (9AM - 12PM) will be used for a structured meeting, and the afternoon session (1PM - 5PM) will be used for a 2 track mini unconference for invitees to the morning session and other interested developers.<br />
<br />
[[PGConf.ASIA2016 Developer Unconference]]<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname) and will be attending:<br />
<br />
* Joe Conway<br />
* Etsuro Fujita<br />
* Magnus Hagander<br />
* Kyotaro Horiguchi<br />
* Kohei KaiGai<br />
* Bruce Momjian<br />
* Dave Page<br />
* Michael Paquier<br />
* Simon Riggs<br />
* Masahiko Sawada<br />
* Teodor Sigaev<br />
* Tomas Vondra<br />
* Amit Kapila<br />
* Takayuki Tsunakawa<br />
<br />
==Agenda==<br />
<br />
TBD<br />
<br />
==Agenda Items==<br />
<br />
Please list any agenda items below for inclusion on the schedule.<br />
* 10.0 Release Schedule<br />
* Providing information for applications which support PostgreSQL<br />
* The future of built-in Postgres sharding<br />
* Revisit varlena for larger data support (>1GB)</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgConf.Asia_2016_Developer_Meeting&diff=28665PgConf.Asia 2016 Developer Meeting2016-11-18T12:11:51Z<p>Kaigai: /* Agenda Items */</p>
<hr />
<div>A meeting of the interested PostgreSQL developers is being planned for the morning of Thursday 1st December, 2016 in Tokyo, prior to PGConf.Asia 2016. In order to keep the numbers manageable, this meeting is by '''invitation only'''. Unfortunately it is quite possible that we've overlooked important individuals during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).<br />
<br />
Please note that the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.6 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies.<br />
<br />
The afternoon will be a Developer Unconference, open to a wider audience.<br />
<br />
This is a PostgreSQL Community event.<br />
<br />
== Meeting Goals ==<br />
<br />
* Review the progress of the 10.0 schedule, and formulate plans to address any issues<br />
* Address any proposed timing, policy, or procedure issues<br />
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br />
<br />
== Time & Location ==<br />
<br />
The event will be held on the fifth floor (using American/Japanese style counting) in room 5A at:<br />
<br />
Akihabara Convention Hall<br />
Akihabara Dai Bldig 4F 1-18-13 Sotokanda, <br />
Chiyoda-ku,<br />
Tokyo 101-0021, <br />
Japan<br />
<br />
Please see the [http://www.akibahall.jp/data/access_eng.html website] for details of how to reach the hall.<br />
<br />
The morning session (9AM - 12PM) will be used for a structured meeting, and the afternoon session (1PM - 5PM) will be used for a 2 track mini unconference for invitees to the morning session and other interested developers.<br />
<br />
[[PGConf.ASIA2016 Developer Unconference]]<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname) and will be attending:<br />
<br />
* Joe Conway<br />
* Etsuro Fujita<br />
* Magnus Hagander<br />
* Kyotaro Horiguchi<br />
* Kohei KaiGai<br />
* Bruce Momjian<br />
* Dave Page<br />
* Michael Paquier<br />
* Simon Riggs<br />
* Masahiko Sawada<br />
* Teodor Sigaev<br />
* Tomas Vondra<br />
* Amit Kapila<br />
<br />
==Agenda==<br />
<br />
TBD<br />
<br />
==Agenda Items==<br />
<br />
Please list any agenda items below for inclusion on the schedule.<br />
* 10.0 Release Schedule<br />
* Providing information for applications which support PostgreSQL</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGStrom&diff=28345PGStrom2016-10-16T01:02:30Z<p>Kaigai: </p>
<hr />
<div>'''IMPORTANT NOTICE'''<br />
<br />
Description of this wikipage is not maintained for more than a year.<br />
Please reference the online manual of the PG-Strom project instead.<br />
<br />
PG-Strom documentation: [http://strom.kaigai.gr.jp/manual.html http://strom.kaigai.gr.jp/manual.html]<br />
<br />
<br />
= Overview =<br />
<br />
PG-Strom is an extension of PostgreSQL, works as custom-scan provider. It was designed to off-load several CPU intensive workloads to GPU devices, to utilize its massive parallel execution capability.<br />
GPU has advantages on number of processor cores (usually several hundreds - thousands) and wider RAM bandwidth (usually multiple times larger capacity than CPU). It works most efficiently when it processes massive amount of numerical operations simultaneously.<br />
<br />
PG-Strom stands on two major ideas:<br />
# Native GPU code generation on the fly<br />
# Asynchronous and pipelined execution model<br />
<br />
PG-Strom checks whether (a part of) given query is executable on GPU devices during query optimization phase. Once it determines the query is available to off-load, PG-Strom construct a source code of GPU native binary on the fly, then kicks just-in-time compile process on the head of execution stage.<br />
<br />
Next, PG-Strom loads a bunch of fetched rows on DMA buffers (15MB per buffer, in the default), then kick DMA transfer and GPU kernel execution in asynchronous manner. CUDA platform allows to handle these tasks in background, so PostgreSQL can make advance the current process.<br />
This asynchronous and in-relation sharding also hides a usual latency around GPU acceleration.<br />
<br />
[[File:PGStrom_Fig_ArchOverview.png]]<br />
<br />
= How to use =<br />
<br />
== Installation ==<br />
<br />
=== Requirements ===<br />
<br />
* PostgreSQL 9.5devel<br />
* [http://developer.nvidia.com/cuda-downloads CUDA 7.0] or later<br />
* x86_64 Linux platform supported by CUDA<br />
<br />
=== Manual Installation ===<br />
<br />
;1. Install CUDA Toolkit v7.0 (or later)<br />
<br />
* You can get CUDA Toolkit from [http://developer.nvidia.com/cuda-downloads NVIDIA], then install the toolkit. Its default installation path is <tt>/usr/local/cuda</tt>. Please don't change the default path.<br />
* For runtime library lookup, you should add a configuration to tell location of the <tt>libnvrtc.so</tt>.<br />
<br />
$ sudo bash<br />
# echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda-lib64.conf<br />
<br />
;2. Install PostgreSQL v9.5devel (or later)<br />
<br />
* You can check out PostgreSQL v9.5devel from [[http://github.com/postgres/postgres GitHUB]].<br />
* Run <tt>./configure</tt> script. At this moment, we recommend to attach <tt>--enable-debug</tt> and <tt>--enable-cassert</tt>.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/postgres/postgres.git pgsql<br />
$ cd pgsql<br />
$ ./configure --enable-debug --enable-cassert<br />
$ make<br />
$ sudo make install<br />
<br />
;3. Install latest PG-Strom<br />
<br />
* You can check out the latest PG-Strom from [[https://github.com/pg-strom/devel.git GitHUB]]<br />
* Ensure <tt>pg_config</tt> is in your command search path.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/pg-strom/devel pg_strom<br />
$ cd pg_strom<br />
$ which pg_config<br />
/usr/local/pgsql/bin/pg_config<br />
$ make<br />
$ sudo make install<br />
<br />
;4. Create database<br />
<br />
* Create database using <tt>initdb</tt>.<br />
* For text comparison, we recommend to add <tt>--no-local</tt> option, because collation aware text comparison is not supported on GPU side.<br />
* Edit <tt>$PGDATA/postgresql.conf</tt> for parameter configuration.<br />
** <tt>$libdir/pg_strom</tt> has to be added to <tt>shared_preload_libraries</tt>.<br />
** <tt>shared_buffers</tt> has to be expanded according to the sizing guideline.<br />
** For other configurations, see the sizing guideline.<br />
<br />
;5. Start/Restart PostgreSQL<br />
<br />
* Start PostgreSQL, using <tt>pg_ctl</tt>. You may see the following message on log.<br />
<br />
LOG: CUDA Runtime version 7.0.0<br />
LOG: NVIDIA driver version: 346.46<br />
LOG: GPU0 GeForce GTX 750 Ti (640 CUDA cores, 1110MHz), L2 2048KB, RAM 2047MB (128bits, 2700KHz), capability 5.0<br />
LOG: NVRTC - CUDA Runtime Compilation vertion 7.0<br />
LOG: database system was shut down at 2015-06-28 17:26:03 JST<br />
LOG: database system is ready to accept connections<br />
LOG: autovacuum launcher started<br />
<br />
;6. Create PG-Strom extension<br />
<br />
* Run <tt>CREATE EXTENSION pg_strom</tt> to setup relevant SQL objects.<br />
<br />
=== Cloud Installation ===<br />
<br />
Under construction<br />
<br />
== Run SQL on GPU ==<br />
<br />
Once PG-Strom is loaded, no special indication is needed to run SQL on GPU device.<br />
It perform as a custom-scan provider of PostgreSQL, and offers alternative scan/join logic that can run on GPU device. If it is feasible and reasonable from estimated cost perspective, planner put custom-scan node instead of the built-in query execution logic.<br />
Below is an ideal case to execute, because size of all the inner relations to be joined is available to load onto GPU RAM at once, and pre-aggregation can reduce number of rows to be processed by CPU effectively.<br />
<br />
postgres=# EXPLAIN SELECT cat, avg(x) FROM t0 natural join t1 natural join t2 natural join t3 natural join t4 GROUP BY cat;<br />
QUERY PLAN<br />
------------------------------------------------------------------------------------------------------------<br />
HashAggregate (cost=300706.95..300707.28 rows=26 width=12)<br />
Group Key: t0.cat<br />
-> Custom Scan (GpuPreAgg) (cost=8860.76..252963.69 rows=1077 width=52)<br />
Bulkload: On (density: 99.00%)<br />
Reduction: Local + Global<br />
-> Custom Scan (GpuJoin) (cost=7860.76..251210.47 rows=9899296 width=12)<br />
Bulkload: On (density: 100.00%)<br />
Depth 1: Logic: GpuHashJoin, HashKeys: (cid), JoinQual: (cid = cid), nrows_ratio: 0.98995197<br />
Depth 2: Logic: GpuHashJoin, HashKeys: (aid), JoinQual: (aid = aid), nrows_ratio: 1.00000000<br />
Depth 3: Logic: GpuHashJoin, HashKeys: (bid), JoinQual: (bid = bid), nrows_ratio: 1.00000000<br />
Depth 4: Logic: GpuHashJoin, HashKeys: (did), JoinQual: (did = did), nrows_ratio: 1.00000000<br />
-> Custom Scan (BulkScan) on t0 (cost=0.00..242855.74 rows=9999774 width=28)<br />
-> Seq Scan on t3 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t1 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t2 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t4 (cost=0.00..734.00 rows=40000 width=4)<br />
(16 rows)<br />
<br />
Below is a set of benchmark result using above microbenchmark workloads according to increase number of tables to be joined.<br />
<br />
[[File:PGStrom Fig MicroBenchGpuJoin.png|512px]]<br />
<br />
;Target Query<br />
SELECT cat, AVG(x) FROM t0 NATURAL JOIN t1 [, ...] GROUP BY cat;<br />
:<tt>t0</tt> contains 100M rows, other relations (<tt>t1</tt>-<tt>t10</tt>) contain 100K rows for each.<br />
:All the data was preloaded.<br />
<br />
;measurement environment<br />
:H/W: NEC Express5800 HR120b-1<br />
:CPU: Xeon E5-2640 (2.50GHz, 6Core)<br />
:RAM: 256GB<br />
:GPU: NVIDIA GTX980<br />
:S/W: PostgreSQL 9.5devel + PG-Strom (26-Mar-2015 ver), CUDA 7.0(x86_64)<br />
<br />
== Data types ==<br />
* Numeric<br />
** smallint<br />
** integer<br />
** bigint<br />
** real<br />
** float<br />
** numeric<br />
* Date and Time<br />
** date<br />
** time<br />
** timestamp<br />
** timestamptz<br />
* Variable length<br />
** Text<br />
** bpchar()<br />
<br />
== Functions and Operators ==<br />
<br />
Under construction<br />
<br />
== GUC options ==<br />
<br />
; pg_strom.enabled<br />
: type: boolean<br />
: default: on<br />
:It enables / disables the entire PG-Strom functionality. Once it gets disabled, CustomScan node never appear in the query execution plan.<br />
<br />
; pg_strom.perfmon<br />
: type: boolean<br />
: default: off<br />
:It enables / disables the performance monitor feature. Once this parameter gets enabled, EXPLAIN ANALYZE will show extra performance information collected from CUDA runtime.<br />
<br />
;pg_strom.cuda_visible_devices<br />
: type: text<br />
: default: NULL<br />
: It allows to configure the variable to be assigned at CUDA_VISIBLE_DEVICES environment variable on system startup time. CUDA runtime ignores the GPU devices, if this environment variable is valid but device number was not lited.<br />
: See [http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars CUDA Environment Variables] for more details.<br />
<br />
;pg_strom.enable_gpuscan<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuScan. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpuhashjoin<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuHashJoin. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpunestloop<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuNestLoop. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpupreagg<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuPreAgg. Once it got disabled, PG-Strom never inject GpuPreAgg node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpusort<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuSort. Once it got disabled, PG-Strom never inject GpuSort node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.chunk_size<br />
: type: integer<br />
: default: 15MB<br />
: The default size of individual DMA buffer (usually called chunk).<br />
<br />
; pg_strom.gpu_setup_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the average cost to set up GPU devices once per query execution. It implies the cost to initialize CUDA context and build (if not on cache) the GPU kernel source<br />
<br />
; pg_strom.gpu_operator_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each operator or function call on GPU devices<br />
<br />
; pg_strom.gpu_tuple_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each tuple (row)<br />
<br />
; pg_strom.program_cache_size<br />
: type: integer<br />
: default: 16MB<br />
: Size of CUDA program cache to be allocated on the static shared memory, during startup time once. So, too large configuration makes waste of RAM. PG-Strom caches a pair of GPU kernel source and CUDA module built, to skip run-time module build second time or later. The cached entry is kept in this area, and too small configuration makes performance degradation because of too frequent run-time compile.<br />
<br />
; pg_strom.bulkload_enabled<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the bulkload feature. If both of parent and child node of the plan tree are CustomScan node managed by PG-Strom, it checks whether bulkload mode is available to exchange the bunck of data between them.<br />
: Once PG-Strom thought bulkload is more reasonable, the child node setup its results on DMA buffer then passes this chunk to the parent node. It allows to reduce waste of CPU cycles for unnecessary data copy between normal memory area and DMA buffers. So, it is the fastest path if we can try.<br />
<br />
; pg_strom.bulkload_density<br />
: type: float<br />
: default: 0.50<br />
: It sets a threshold to use bulkloading mode to exchange a bunch of data from the underlying node of the plan tree. If child node may increase / reduce number of tuples more / less than the threshold, we don't apply bulkload mode even if possible.<br />
<br />
; pg_strom.max_async_tasks<br />
: type: integer<br />
: default: 32<br />
: It sets maximum number of asynchronous tasks that are sum of launched and pending. When GPU kernel execution or GPU kernel build (usual case) takes times, PG-Strom makes advance child node scanning and loading to DMA buffer. This parameter works as threshold of this prefetching.<br />
<br />
; pg_strom.max_workers<br />
: type: integer<br />
: default: ---<br />
: It sets maximum number of background worker process if and when PG-Strom also takes CPU parallel execution.<br />
<br />
== Sizing Guideline ==<br />
<br />
Under construction<br />
<br />
<br />
= Background =<br />
<br />
== Parallel Approach ==<br />
<br />
We have a few ways to enhance performance of PostgreSQL.<br />
* Homogeneous scale-up<br />
* Heterogeneous scale-up<br />
* Scale-out<br />
PG-Strom takes heterogeneous scale-up approach; that utilizes the most advantaged hardware according to the characteristics of the workloads. In other words, it assigns simple but massive amount of numerical calculation, that is too previous to run on CPU cores, on GPU devices.<br />
<br />
[[File:PGStrom_Fig_ParallelApproach.png]]<br />
<br />
= Future Development =<br />
<br />
== PostgreSQL core ==<br />
<br />
== PG-Strom ==<br />
<br />
== Long term development ==</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGStrom_Roadmap&diff=27950PGStrom Roadmap2016-08-03T09:23:20Z<p>Kaigai: Created page with "= PostgreSQL 9.6 timeline = == PG-Strom extension == * '''PL/CUDA''' (done) ** It allows to write up raw CUDA code as SQL function. PG-Strom moves bunch of data set to GPU th..."</p>
<hr />
<div>= PostgreSQL 9.6 timeline =<br />
<br />
== PG-Strom extension ==<br />
* '''PL/CUDA''' (done)<br />
** It allows to write up raw CUDA code as SQL function. PG-Strom moves bunch of data set to GPU then kicks user defined kernel function.<br />
* '''CPU+GPU Hybrid Parallel''' (in-progress)<br />
** It makes CustomScan nodes under the Gather node. Each worker uses GPU for finer grained parallel execution.<br />
* '''SSD-to-GPU Direct DMA''' (in-progress)<br />
** It enables direct data transfer from SSD blocks to GPU RAM, then CPU receives only valid rows, joined records, or partially aggregated values.<br />
<br />
= PostgreSQL 10 timeline =<br />
<br />
== In-database Analytics ==<br />
* '''Sparse Matrix Datatype''' - It tries to represent vector/matrix data as a basis of analytic/statistical algorithm on database system. It intends (1) large amount of data set more than 1GB, and (2) reasonable memory consumption if matrix is actually sparse.<br />
* '''Analytic Functions''' - as a use case of sparse matrix, we plan to submit several analytic/statistical functions as a contrib module. Unsupervized learning algorithms (kmeans, Ward-clustering) are candidates.<br />
<br />
== CPU Parallelism ==<br />
* '''Parallel aware Append''' - In case when multiple partitioned tables are scanned or simple UNION ALL conjunction, we have good chance to run these portions in parallel under the Gather node. It is a feature 0racle does not have but people often wants. <br />
<br />
== CustomScan Interface ==<br />
* '''Improvement''' - Continuous improvement to follow up planner / executor enhancement. The goal of CustomScan is allowing extensions to implement arbitrary logic as if it is built-in feature.<br />
* '''Limit support''' - pass_down_bound() tells underlying nodes how much rows are required to produce if it is Sort, MergeAppend or Result. If a sorting logic is implemented on top of CustomScan, it cannot see the hint information.<br />
<br />
= Beyond PostgreSQL 10 =<br />
<br />
Right now, I don't have enough time working on the topic below, even motivated.<br />
<br />
== Wise optimization for OLAP ==<br />
* '''Risk factor in nrows estimation'''<br />
** Execution cost of nested-loop growth rapidly if number of outer rows is much larger than the estimation. Right now, optimizer relies on the estimated nrows, however, its exactness fully depends on complicity of the underlying sub-plan. If outer plan is simple table scan, its estimated nrows are almost correct. So, its risk factor shall be small. On the other hands, if outer plan is multi-tables join with complicated qualifiers, estimation is less reliable.<br />
** For OLAP workloads, we have to pay attention to variation of the estimated nrows. It may needs to swith more robust algorithm (HashJoin, MergeJoin) when estimated nrows is less reliable.<br />
* '''Group by before JOIN'''<br />
** Under some condition, we can rewrite query to place GROUP BY before join. It will reduce number of rows to be joinned, and allows to utilize massive parallel processors more efficient.<br />
** Earlier study: http://www.comp.nus.edu.sg/~cs5226/papers/groupby-join-icde94.pdf<br />
* '''Wise optimization on some corner cases'''<br />
** In some TPC-DS workloads, parametalized sub-query makes massive performance degradation, even though we can rewrite most of the problematic queries in mechanical way. And, 0racle pulls-up these queries correctly.</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PostgreSQL_Conference_Europe_Talks_2015&diff=26248PostgreSQL Conference Europe Talks 20152015-10-31T05:17:52Z<p>Kaigai: KaiGai's slides</p>
<hr />
<div><br />
== Tuesday (Training) ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Trainer||Title (Link to slides)<br />
|+<br />
| || || || <br />
|+<br />
|}<br />
<br />
== Wednesday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 12:10 - 13:00 || Ballroom A+B || Joe Conway || [http://www.joeconway.com/presentations/text_search-pgconfeu2015.pdf Where’s Waldo? - Text Search and Pattern Matching in PostgreSQL]||<br />
|+<br />
| 12:10 - 13:00 || Ballroom C+D || Mladen Marinović || [https://bitbucket.org/marin/pgconfeu2015/raw/883632334c68663e10ab9637003f32bc2fadb279/Dockerizing%20a%20Largrer%20PostgreSQL%20Installation.pdf Dockerizing a Larger PostgreSQL Installation: What could possibly go wrong?]||<br />
|+<br />
| 12:10 - 13:00 || Palais I-III || Nikolay Shaplov || [https://github.com/dhyannataraj/tuple-internals-presentation Tuple internals: exposing, exploring and explaining]||<br />
|+<br />
| 14:00 - 14:50 || Ballroom C+D || Kaarel Moppel || [https://docs.google.com/presentation/d/1BzzhJKHsie3cqjJwYa0y2QeTzMDsJtanbtsqq4As5tk/edit?usp=sharing A PostgreSQL DBA's toolbelt]|| <br />
|+<br />
| 14:00 - 14:50 || Ballroom A+B || Victor Blomqvist || [https://onedrive.live.com/redir?resid=AD5D909BDDBF9E98!34117&authkey=!AJQGy3wzRaFd_wo&ithint=file%2cpdf Location based dating in China - 0 to 100000000 daily swipes in a year] ||<br />
|+<br />
| 15:00 - 15:50 || Ballroom A+B || Michael Banck || [[Media: Dumping_the_mainframe_mbanck.pdf|Dumping the Mainframe: Migration Study from DB2 UDB to PostgreSQL]]||<br />
|+<br />
|}<br />
<br />
== Thursday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 9:30 || Ballroom A+B || Michael Paquier || [http://michael.otacoo.com/content/materials/20150917_pgopen2015_standbys.pdf WAL, Standbys and Postgrs 9.5]<br />
|+<br />
| 11:50 || Ballroom A+B || Balázs Bárány || [https://datascientist.at/wp-content/uploads/2015/01/B%C3%A1r%C3%A1ny-Data-Science-with-PostgreSQL.pdf Data Science with PostgreSQL]<br />
|+<br />
| 12:50 || Palais I-III || Jim Mlodgenski || [http://www.slideshare.net/jim_mlodgenski/an-introduction-to-postresql-triggers An Introduction To PostgreSQL Triggers]<br />
|+<br />
|}<br />
<br />
== Friday ==<br />
<br />
{| class="wikitable"<br />
|Time||Room||Speaker||Title (Link to slides)<br />
|+<br />
| 09:30 || Ballroom C+D || Jim Mlodgenski || [http://www.slideshare.net/jim_mlodgenski/debugging-your-plpgsql-code Debugging Your PL/pgSQL Code ]<br />
|+<br />
| 09:30 || Palais I-III || KaiGai Kohei || [http://www.slideshare.net/kaigai/gpgpu-accelerates-postgresql-unlock-the-power-of-multithousand-cores GPGPU Accelerates PostgreSQL - Unlock the power of multi-thousand cores]<br />
|+<br />
| 13:40 || Ballroom A+B || Vincent Picavet || [https://github.com/Oslandia/presentations/raw/master/pgconf_eu_2015/beyond_postgis_basics.pdf Beyond PostGIS basics : more spatial ! ]<br />
|+<br />
| 13:40 || Palais I-III || Heikki Linnakangas || [http://hlinnaka.iki.fi/presentations/Index-internals-Vienna2015.pdf Index Internals ]<br />
|}</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGStrom&diff=25342PGStrom2015-06-29T02:01:51Z<p>Kaigai: /* Run SQL on GPU */</p>
<hr />
<div>= Overview =<br />
<br />
PG-Strom is an extension of PostgreSQL, works as custom-scan provider. It was designed to off-loads several CPU intensive workloads to GPU devices, to utilize its massive parallel execution capability.<br />
GPU has advantages on number of processor cores (usually several hundreds - thausands) and wider bandwidth of RAM (usually multiple times larger capacity than CPU). It works most effeciently when it processes massive amount of numerical operations simultaneously.<br />
<br />
PG-Strom stands on two major ideas:<br />
# Native GPU code generation on the fly<br />
# Asynchronous and pipelined execution model<br />
<br />
PG-Strom checks whether (a part of) given query is executable on GPU devices during query optimization phase. Once it determines the query is available to off-load, PG-Strom construct a source code of GPU native binary on the fly, then kicks just-in-time compile process on the head of execution stage.<br />
<br />
Next, PG-Strom loads a bunch of fetched rows on DMA buffers (15MB per buffer, in the default), then kick DMA transfer and GPU kernel execution in asynchronous manner. CUDA platform allows to handle these tasks in background, so PostgreSQL can make advance the current process.<br />
This asynchronous and in-relation sharding also hides a usual latency around GPU acceleration.<br />
<br />
[[File:PGStrom_Fig_ArchOverview.png]]<br />
<br />
<br />
= How to use =<br />
<br />
== Installation ==<br />
<br />
=== Requirements ===<br />
<br />
* PostgreSQL 9.5devel<br />
* [http://developer.nvidia.com/cuda-downloads CUDA 7.0] or later<br />
* x86_64 Linux platform supported by CUDA<br />
<br />
=== Manual Installation ===<br />
<br />
;1. Install CUDA Toolkit v7.0 (or later)<br />
<br />
* You can get CUDA Toolkit from [http://developer.nvidia.com/cuda-downloads NVIDIA], then install the toolkit. Its default installation path is <tt>/usr/local/cuda</tt>. Please don't change the default path.<br />
* For runtime library lookup, you should add a configuration to tell location of the <tt>libnvrtc.so</tt>.<br />
<br />
$ sudo bash<br />
# echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda-lib64.conf<br />
<br />
;2. Install PostgreSQL v9.5devel (or later)<br />
<br />
* You can check out PostgreSQL v9.5devel from [[http://github.com/postgres/postgres GitHUB]].<br />
* Run <tt>./configure</tt> script. At this moment, we recommend to attach <tt>--enable-debug</tt> and <tt>--enable-cassert</tt>.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/postgres/postgres.git pgsql<br />
$ cd pgsql<br />
$ ./configure --enable-debug --enable-cassert<br />
$ make<br />
$ sudo make install<br />
<br />
;3. Install latest PG-Strom<br />
<br />
* You can check out the latest PG-Strom from [[https://github.com/pg-strom/devel.git GitHUB]]<br />
* Ensure <tt>pg_config</tt> is in your command search path.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/pg-strom/devel pg_strom<br />
$ cd pg_strom<br />
$ which pg_config<br />
/usr/local/pgsql/bin/pg_config<br />
$ make<br />
$ sudo make install<br />
<br />
;4. Create datanase<br />
<br />
* Create database using <tt>initdb</tt>.<br />
* For text comparison, we recommend to add <tt>--no-local</tt> option, because collation aware text comparison is not supported on GPU side.<br />
* Edit <tt>$PGDATA/postgresql.conf</tt> for parameter configuration.<br />
** <tt>$libdir/pg_strom</tt> has to be added to <tt>shared_preload_libraries</tt>.<br />
** <tt>shared_buffers</tt> has to be expanded according to the sizing guideline.<br />
** For other configurations, see the sizing guideline.<br />
<br />
;5. Start/Restart PostgreSQL<br />
<br />
* Start PostgreSQL, using <tt>pg_ctl</tt>. You may see the following message on log.<br />
<br />
LOG: CUDA Runtime version 7.0.0<br />
LOG: NVIDIA driver version: 346.46<br />
LOG: GPU0 GeForce GTX 750 Ti (640 CUDA cores, 1110MHz), L2 2048KB, RAM 2047MB (128bits, 2700KHz), capability 5.0<br />
LOG: NVRTC - CUDA Runtime Compilation vertion 7.0<br />
LOG: database system was shut down at 2015-06-28 17:26:03 JST<br />
LOG: database system is ready to accept connections<br />
LOG: autovacuum launcher started<br />
<br />
;6. Create PG-Strom extension<br />
<br />
* Run <tt>CREATE EXTENSION pg_strom</tt> to setup relevant SQL objects.<br />
<br />
=== Cloud Installation ===<br />
<br />
Under construction<br />
<br />
== Run SQL on GPU ==<br />
<br />
Once PG-Strom is loaded, no special indication is needed to run SQL on GPU device.<br />
It perform as a custom-scan provider of PostgreSQL, and offers alternative scan/join logic that can run on GPU device. If it is feasible and reasonable from estimated cost perspective, planner put custom-scan node instead of the built-in query execution logic.<br />
Below is an ideal case to execute, because size of all the inner relations to be joined is available to load onto GPU RAM at once, and pre-aggregation can reduce number of rows to be processed by CPU effectively.<br />
<br />
postgres=# EXPLAIN SELECT cat, avg(x) FROM t0 natural join t1 natural join t2 natural join t3 natural join t4 GROUP BY cat;<br />
QUERY PLAN<br />
------------------------------------------------------------------------------------------------------------<br />
HashAggregate (cost=300706.95..300707.28 rows=26 width=12)<br />
Group Key: t0.cat<br />
-> Custom Scan (GpuPreAgg) (cost=8860.76..252963.69 rows=1077 width=52)<br />
Bulkload: On (density: 99.00%)<br />
Reduction: Local + Global<br />
-> Custom Scan (GpuJoin) (cost=7860.76..251210.47 rows=9899296 width=12)<br />
Bulkload: On (density: 100.00%)<br />
Depth 1: Logic: GpuHashJoin, HashKeys: (cid), JoinQual: (cid = cid), nrows_ratio: 0.98995197<br />
Depth 2: Logic: GpuHashJoin, HashKeys: (aid), JoinQual: (aid = aid), nrows_ratio: 1.00000000<br />
Depth 3: Logic: GpuHashJoin, HashKeys: (bid), JoinQual: (bid = bid), nrows_ratio: 1.00000000<br />
Depth 4: Logic: GpuHashJoin, HashKeys: (did), JoinQual: (did = did), nrows_ratio: 1.00000000<br />
-> Custom Scan (BulkScan) on t0 (cost=0.00..242855.74 rows=9999774 width=28)<br />
-> Seq Scan on t3 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t1 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t2 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t4 (cost=0.00..734.00 rows=40000 width=4)<br />
(16 rows)<br />
<br />
Below is a set of benchmark result using above microbenchmark workloads according to increase number of tables to be joined.<br />
<br />
[[File:PGStrom Fig MicroBenchGpuJoin.png|512px]]<br />
<br />
;Target Query<br />
SELECT cat, AVG(x) FROM t0 NATURAL JOIN t1 [, ...] GROUP BY cat;<br />
:<tt>t0</tt> contains 100M rows, other relations (<tt>t1</tt>-<tt>t10</tt>) contain 100K rows for each.<br />
:All the data was preloaded.<br />
<br />
;measurement environment<br />
:H/W: NEC Express5800 HR120b-1<br />
:CPU: Xeon E5-2640 (2.50GHz, 6Core)<br />
:RAM: 256GB<br />
:GPU: NVIDIA GTX980<br />
:S/W: PostgreSQL 9.5devel + PG-Strom (26-Mar-2015 ver), CUDA 7.0(x86_64)<br />
<br />
== Data types ==<br />
* Numeric<br />
** smallint<br />
** integer<br />
** bigint<br />
** real<br />
** float<br />
** numeric<br />
* Date and Time<br />
** date<br />
** time<br />
** timestamp<br />
** timestamptz<br />
* Variable length<br />
** Text<br />
** bpchar()<br />
<br />
== Functions and Operators ==<br />
<br />
Under construction<br />
<br />
== GUC options ==<br />
<br />
; pg_strom.enabled<br />
: type: boolean<br />
: default: on<br />
:It enables / disables the entire PG-Strom functionality. Once it gets disabled, CustomScan node never appear in the query execution plan.<br />
<br />
; pg_strom.perfmon<br />
: type: boolean<br />
: default: off<br />
:It enables / disables the performance monitor feature. Once this parameter gets enabled, EXPLAIN ANALYZE will show extra performance information collected from CUDA runtime.<br />
<br />
;pg_strom.cuda_visible_devices<br />
: type: text<br />
: default: NULL<br />
: It allows to configure the variable to be assigned at CUDA_VISIBLE_DEVICES environment variable on system startup time. CUDA runtime ignores the GPU devices, if this environment variable is valid but device number was not lited.<br />
: See [http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars CUDA Environment Variables] for more details.<br />
<br />
;pg_strom.enable_gpuscan<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuScan. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpuhashjoin<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuHashJoin. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpunestloop<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuNestLoop. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpupreagg<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuPreAgg. Once it got disabled, PG-Strom never inject GpuPreAgg node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpusort<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuSort. Once it got disabled, PG-Strom never inject GpuSort node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.chunk_size<br />
: type: integer<br />
: default: 15MB<br />
: The default size of individual DMA buffer (usually called chunk).<br />
<br />
; pg_strom.gpu_setup_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the average cost to set up GPU devices once per query execution. It implies the cost to initialize CUDA context and build (if not on cache) the GPU kernel source<br />
<br />
; pg_strom.gpu_operator_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each operator or function call on GPU devices<br />
<br />
; pg_strom.gpu_tuple_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each tuple (row)<br />
<br />
; pg_strom.program_cache_size<br />
: type: integer<br />
: default: 16MB<br />
: Size of CUDA program cache to be allocated on the static shared memory, during startup time once. So, too large configuration makes waste of RAM. PG-Strom caches a pair of GPU kernel source and CUDA module built, to skip run-time module build second time or later. The cached entry is kept in this area, and too small configuration makes performance degradation because of too frequent run-time compile.<br />
<br />
; pg_strom.bulkload_enabled<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the bulkload feature. If both of parent and child node of the plan tree are CustomScan node managed by PG-Strom, it checks whether bulkload mode is available to exchange the bunck of data between them.<br />
: Once PG-Strom thought bulkload is more reasonable, the child node setup its results on DMA buffer then passes this chunk to the parent node. It allows to reduce waste of CPU cycles for unnecessary data copy between normal memory area and DMA buffers. So, it is the fastest path if we can try.<br />
<br />
; pg_strom.bulkload_density<br />
: type: float<br />
: default: 0.50<br />
: It sets a threshold to use bulkloading mode to exchange a bunch of data from the underlying node of the plan tree. If child node may increase / reduce number of tuples more / less than the threshold, we don't apply bulkload mode even if possible.<br />
<br />
; pg_strom.max_async_tasks<br />
: type: integer<br />
: default: 32<br />
: It sets maximum number of asynchronous tasks that are sum of launched and pending. When GPU kernel execution or GPU kernel build (usual case) takes times, PG-Strom makes advance child node scanning and loading to DMA buffer. This parameter works as threshold of this prefetching.<br />
<br />
; pg_strom.max_workers<br />
: type: integer<br />
: default: ---<br />
: It sets maximum number of background worker process if and when PG-Strom also takes CPU parallel execution.<br />
<br />
== Sizing Guideline ==<br />
<br />
Under construction<br />
<br />
<br />
= Background =<br />
<br />
== Parallel Approach ==<br />
<br />
We have a few ways to enhance performance of PostgreSQL.<br />
* Homogeneous scale-up<br />
* Heterogeneous scale-up<br />
* Scale-out<br />
PG-Strom takes heterogeneous scale-up approach; that utilizes the most advantaged hardware according to the characteristics of the workloads. In other words, it assigns simple but massive amount of numerical calculation, that is too previous to run on CPU cores, on GPU devices.<br />
<br />
[[File:PGStrom_Fig_ParallelApproach.png]]<br />
<br />
= Future Development =<br />
<br />
== PostgreSQL core ==<br />
<br />
== PG-Strom ==<br />
<br />
== Long term development ==</div>Kaigaihttps://wiki.postgresql.org/index.php?title=What%27s_new_in_PostgreSQL_9.5&diff=25336What's new in PostgreSQL 9.52015-06-28T10:57:44Z<p>Kaigai: </p>
<hr />
<div>(This page is currently under development ahead of the release of PostgreSQL 9.5)<br />
<br />
This page contains an overview of PostgreSQL Version 9.5's features, including descriptions, testing and usage information, and links to blog posts containing further information. See also [[PostgreSQL 9.5 Open Items]].<br />
<br />
=Major new features=<br />
<br />
==IMPORT FOREIGN SCHEMA==<br />
<br />
Previously, in order to create a foreign table in PostgreSQL, you would need to define the table, referencing the destination columns and data types, and if you have a lot of tables, this can become tedious and error-prone, and when those tables change, you need to do it all over again...<br />
<br />
'''CREATE FOREIGN TABLE remote.customers ('''<br />
'''id int NOT NULL,'''<br />
'''name text,'''<br />
'''company text,'''<br />
'''registered_date date,'''<br />
'''expiry_date date,'''<br />
'''active boolean,'''<br />
'''status text,'''<br />
'''account_level text) SERVER dest_server OPTIONS (schema_name 'public');'''<br />
<br />
'''CREATE FOREIGN TABLE remote.purchases ('''<br />
'''id int NOT NULL,'''<br />
'''purchase_time timestamptz,'''<br />
'''payment_time timestamptz,'''<br />
'''itemid int,'''<br />
'''volume int,'''<br />
'''invoice_sent boolean) SERVER dest_server OPTIONS (schema_name 'public');'''<br />
<br />
As of PostgreSQL 9.5, you can import tables en masse:<br />
<br />
'''IMPORT FOREIGN SCHEMA public'''<br />
'''FROM SERVER dest_server INTO remote;'''<br />
<br />
This would create foreign tables in the schema named "remote" for every table that appeared in the public schema on the foreign server labelled "dest_server".<br />
<br />
You can also filter out any tables you don't wish:<br />
<br />
IMPORT FOREIGN SCHEMA public<br />
'''EXCEPT (reports, audit)'''<br />
FROM SERVER dest_server INTO remote;<br />
<br />
Or limit it to just a specific set of tables:<br />
<br />
IMPORT FOREIGN SCHEMA public<br />
'''LIMIT TO (customers, purchases)'''<br />
FROM SERVER dest_server INTO remote;<br />
<br />
====Links====<br />
[http://www.depesz.com/2014/07/14/waiting-for-9-5-implement-import-foreign-schema/ Waiting for 9.5 – Implement IMPORT FOREIGN SCHEMA.]<br />
<br />
[http://blog.2ndquadrant.com/postgresql-9-5-import-foreign-schema/ PostgreSQL 9.5: IMPORT FOREIGN SCHEMA]<br />
<br />
==Row-Level Security Policies==<br />
<br />
Additional security can be added to tables to prevent users from accessing rows they shouldn't be able to see.<br />
<br />
Say you had a table with log data, where the username column contained the database user name which created the log entry:<br />
<br />
'''CREATE TABLE log ('''<br />
'''id serial primary key,'''<br />
'''username text,'''<br />
'''log_event text);'''<br />
<br />
But you don't want users to see the log entries from other users, so we create a policy that says you're allowed to see the row if the username column matches the current user running the query:<br />
<br />
'''CREATE POLICY policy_user_log ON log'''<br />
'''FOR ALL'''<br />
'''TO PUBLIC'''<br />
'''USING (username = current_user);'''<br />
<br />
And then we enable Row Level Security on the table:<br />
<br />
'''ALTER TABLE log'''<br />
'''ENABLE ROW LEVEL SECURITY;'''<br />
<br />
As the user "report", we would then only see rows where the username column contained the value 'report':<br />
<br />
# '''SELECT * FROM log;'''<br />
id | username | log_event <br />
----+----------+----------------<br />
1 | report | DELETE issued<br />
4 | report | Reset accounts<br />
(2 rows)<br />
<br />
As the user "messaging", we see a different set of rows:<br />
<br />
id | username | log_event <br />
----+-----------+----------------------<br />
2 | messaging | Message queue purged<br />
3 | messaging | Reset accounts<br />
(2 rows)<br />
<br />
Whereas the "postgres" user, as the superuser would get:<br />
<br />
id | username | log_event <br />
----+-----------+----------------------<br />
1 | report | DELETE issued<br />
2 | messaging | Message queue purged<br />
3 | messaging | Reset accounts<br />
4 | report | Reset accounts<br />
(4 rows)<br />
<br />
That's because the superuser sees all rows due to the BYPASSRLS attribute on the superuser role by default.<br />
<br />
====Links====<br />
[http://www.depesz.com/2014/10/02/waiting-for-9-5-row-level-security-policies-rls/ Waiting for 9.5 – Row-Level Security Policies (RLS)]<br />
<br />
==BRIN Indexes==<br />
<br />
BRIN stands for Block Range INdexes, and store metadata on a range of pages. At the moment this means the minimum and maximum values per block.<br />
<br />
This results in an inexpensive index that occupies a very small amount of space, and can speed up queries in extremely large tables. This allows the index to determine which blocks are the only ones worth checking, and all others can be skipped. So if a 10GB table of order contained rows that were generally in order of order date, a BRIN index on the order_date column would allow the majority of the table to be skipped rather than performing a full sequential scan. This will still be slower than a regular BTREE index on the same column, but with the benefits of it being far smaller and requires less maintenance.<br />
<br />
For example:<br />
<br />
-- Create the table<br />
'''CREATE TABLE orders ('''<br />
'''id int,'''<br />
'''order_date timestamptz,'''<br />
'''item text);'''<br />
<br />
-- Insert lots of data into it<br />
'''INSERT INTO orders (order_date, item)'''<br />
'''SELECT x, 'dfiojdso' '''<br />
'''FROM generate_series('2000-01-01 00:00:00'::timestamptz, '2015-03-01 00:00:00'::timestamptz,'2 seconds'::interval) a(x);'''<br />
<br />
-- Let's look at how much space the table occupies<br />
# '''\dt+ orders'''<br />
List of relations<br />
Schema | Name | Type | Owner | Size | Description <br />
--------+--------+-------+-------+-------+-------------<br />
public | orders | table | thom | 13 GB | <br />
(1 row)<br />
<br />
-- What's involved in finding the orders between 2 dates<br />
# '''EXPLAIN ANALYSE SELECT count(*) FROM orders WHERE order_date BETWEEN '2012-01-04 09:00:00' and '2014-01-04 14:30:00';'''<br />
QUERY PLAN <br />
-------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Aggregate (cost=5425021.80..5425021.81 rows=1 width=0) (actual time=30172.428..30172.429 rows=1 loops=1)<br />
-> Seq Scan on orders (cost=0.00..5347754.00 rows=30907121 width=0) (actual time=6050.015..28552.976 rows=31589101 loops=1)<br />
Filter: ((order_date >= '2012-01-04 09:00:00+00'::timestamp with time zone) AND (order_date <= '2014-01-04 14:30:00+00'::timestamp with time zone))<br />
Rows Removed by Filter: 207652500<br />
Planning time: 0.140 ms<br />
Execution time: 30172.482 ms<br />
(6 rows)<br />
<br />
-- Now let's create a BRIN index on the order_date column<br />
'''CREATE INDEX idx_order_date_brin<br />
ON orders<br />
USING BRIN (order_date);'''<br />
<br />
-- And see how much space it takes up<br />
# '''\di+ idx_order_date_brin'''<br />
List of relations<br />
Schema | Name | Type | Owner | Table | Size | Description <br />
--------+---------------------+-------+-------+--------+--------+-------------<br />
public | idx_order_date_brin | index | thom | orders | 504 kB | <br />
(1 row)<br />
<br />
-- Now let's see how much faster the query is with this very small index<br />
# '''EXPLAIN ANALYSE SELECT count(*) FROM orders WHERE order_date BETWEEN '2012-01-04 09:00:00' and '2014-01-04 14:30:00';'''<br />
QUERY PLAN <br />
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Aggregate (cost=2616868.60..2616868.61 rows=1 width=0) (actual time=6347.651..6347.651 rows=1 loops=1)<br />
-> Bitmap Heap Scan on orders (cost=316863.99..2539600.80 rows=30907121 width=0) (actual time=36.366..4686.634 rows=31589101 loops=1)<br />
Recheck Cond: ((order_date >= '2012-01-04 09:00:00+00'::timestamp with time zone) AND (order_date <= '2014-01-04 14:30:00+00'::timestamp with time zone))<br />
Rows Removed by Index Recheck: 6419<br />
Heap Blocks: lossy=232320<br />
-> Bitmap Index Scan on idx_order_date_brin (cost=0.00..309137.21 rows=30907121 width=0) (actual time=35.567..35.567 rows=2323200 loops=1)<br />
Index Cond: ((order_date >= '2012-01-04 09:00:00+00'::timestamp with time zone) AND (order_date <= '2014-01-04 14:30:00+00'::timestamp with time zone))<br />
Planning time: 0.108 ms<br />
Execution time: 6347.701 ms<br />
(9 rows)<br />
<br />
This example is on an SSD drive, so the results would be even more pronounced on an HDD.<br />
<br />
By default, the block size is 128 pages. This resolution can be increased or decreased using the pages_per_range:<br />
<br />
-- Create an index with 32 pages per block<br />
CREATE INDEX idx_order_date_brin_32<br />
ON orders<br />
USING BRIN (order_date) '''WITH (pages_per_range = 32)''';<br />
<br />
-- Create an index with 512 pages per block<br />
CREATE INDEX idx_order_date_brin_512<br />
ON orders<br />
USING BRIN (order_date) '''WITH (pages_per_range = 512)''';<br />
<br />
The lower the pages per block, the more space the index will occupy, but the less lossy the index will be, i.e. it will need to discard fewer rows.<br />
<br />
# '''\di+ idx_order_date_brin*'''<br />
List of relations<br />
Schema | Name | Type | Owner | Table | Size | Description <br />
--------+-------------------------+-------+-------+--------+---------+-------------<br />
public | idx_order_date_brin | index | thom | orders | 504 kB | <br />
public | idx_order_date_brin_32 | index | thom | orders | 1872 kB | <br />
public | idx_order_date_brin_512 | index | thom | orders | 152 kB | <br />
(3 rows)<br />
<br />
====Links====<br />
[http://www.depesz.com/2014/11/22/waiting-for-9-5-brin-block-range-indexes/ Waiting for 9.5 – BRIN: Block Range Indexes.]<br />
<br />
==Foreign Table Inheritance==<br />
<br />
Foreign tables can now either inherit local tables, or be inherited from.<br />
<br />
For example, a local table can inherit a foreign table:<br />
<br />
-- Create a new table which inherits from the foreign table<br />
# '''CREATE local_customers () inherits (remote.customers);'''<br />
<br />
-- Insert some data into it<br />
# '''INSERT INTO local_customers VALUES (16, 'Bruce',$$Jo's Cupcakes$$, '2015-01-15', '2017-01-14', true, 'running', 'basic');'''<br />
<br />
-- And if we query the parent foreign table...<br />
# '''SELECT tableoid::regclass, * FROM remote.customers;'''<br />
tableoid | id | name | company | registered_date | expiry_date | active | status | account_level <br />
------------------+----+-------+---------------+-----------------+-------------+--------+---------+---------------<br />
remote.customers | 1 | James | Hughson Corp | 2014-05-03 | 2016-05-02 | t | idle | premium<br />
local_customers | 16 | Bruce | Jo's Cupcakes | 2015-01-15 | 2017-01-14 | t | running | basic<br />
(2 rows)<br />
<br />
Or a foreign table can be made to inherit from a local table:<br />
<br />
-- Create a new table that the foreign table will be a child of<br />
# '''CREATE TABLE master_customers (LIKE remote.customers);'''<br />
<br />
-- Insert a new row into this table<br />
# '''INSERT INTO master_customers VALUES (99, 'Jolly',$$Cineplanet$$, '2014-10-30', '2016-10-29', true, 'running', 'premium');'''<br />
<br />
-- Have the foreign table inherit from the new table<br />
# '''ALTER TABLE remote.customers INHERIT master_customers;'''<br />
<br />
-- Let's have a look at the contents of the new table now<br />
# '''SELECT tableoid::regclass, * FROM master_customers;'''<br />
tableoid | id | name | company | registered_date | expiry_date | active | status | account_level <br />
------------------+----+-------+---------------+-----------------+-------------+--------+---------+---------------<br />
master_customers | 99 | Jolly | Cineplanet | 2014-10-30 | 2016-10-29 | t | running | premium<br />
remote.customers | 1 | James | Hughson Corp | 2014-05-03 | 2016-05-02 | t | idle | premium<br />
local_customers | 16 | Bruce | Jo's Cupcakes | 2015-01-15 | 2017-01-14 | t | running | basic<br />
(3 rows)<br />
<br />
-- And the query plan...<br />
# '''EXPLAIN ANALYSE SELECT tableoid::regclass, * FROM master_customers;'''<br />
QUERY PLAN <br />
---------------------------------------------------------------------------------------------------------------------------<br />
Result (cost=0.00..140.80 rows=1012 width=145) (actual time=0.014..0.595 rows=3 loops=1)<br />
-> Append (cost=0.00..140.80 rows=1012 width=145) (actual time=0.012..0.591 rows=3 loops=1)<br />
-> '''Seq Scan on master_customers''' (cost=0.00..1.48 rows=48 width=145) (actual time=0.012..0.013 rows=1 loops=1)<br />
-> '''Foreign Scan on customers''' (cost=100.00..124.52 rows=484 width=145) (actual time=0.567..0.567 rows=1 loops=1)<br />
-> '''Seq Scan on local_customers''' (cost=0.00..14.80 rows=480 width=145) (actual time=0.007..0.008 rows=1 loops=1)<br />
Planning time: 0.256 ms<br />
Execution time: 1.040 ms<br />
(7 rows)<br />
<br />
====Links====<br />
[http://www.depesz.com/2015/04/02/waiting-for-9-5-allow-foreign-tables-to-participate-in-inheritance/ Waiting for 9.5 – Allow foreign tables to participate in inheritance. – A.K.A. PostgreSQL got sharding.]<br />
<br />
[http://michael.otacoo.com/postgresql-2/postgres-9-5-feature-foreign-table-inheritance/ Postgres 9.5 feature highlight: Scale-out with Foreign Tables now part of Inheritance Trees]<br />
<br />
==GROUPING SETS, CUBE and ROLLUP==<br />
<br />
This set of features allows one to summarise data into sets.<br />
<br />
For example, if we have this data:<br />
<br />
# SELECT * FROM employees;<br />
name | role | department | gender <br />
----------+-----------------+------------+--------<br />
Tim | Manager | Sales | Male<br />
Sarah | Manager | Finance | Female<br />
Neil | Accountant | Finance | Male<br />
Joe | Project Manager | Sales | Male<br />
Yvette | Project Manager | Finance | Female<br />
Craig | Project Manager | IT | Male<br />
Penelope | Manager | IT | Female<br />
(7 rows)<br />
<br />
If we wanted to see summaries for each department, role and gender, we can use GROUPING SETS:<br />
<br />
# SELECT department, role, gender, count(*)<br />
FROM employees<br />
GROUP BY '''GROUPING SETS (department, role, gender, ())''';<br />
<br />
department | role | gender | count <br />
------------+-----------------+-----------+-------<br />
Finance | | | 3<br />
IT | | | 2<br />
Sales | | | 2<br />
| | | 7<br />
| | Female | 3<br />
| | Male | 4<br />
| Accountant | | 1<br />
| Manager | | 3<br />
| Project Manager | | 3<br />
(9 rows)<br />
<br />
<br />
Here we can see the count of employees in each department, each role, and each gender. We also get a total where all columns except count are blank.<br />
<br />
If we wanted a count for every combination of those 3 categories, we could use CUBE:<br />
<br />
# SELECT department, role, gender, count(*)<br />
FROM employees<br />
GROUP BY '''CUBE (department, role, gender)''';<br />
<br />
department | role | gender | count <br />
------------+-----------------+-----------+-------<br />
Finance | Accountant | Male | 1<br />
Finance | Accountant | | 1<br />
Finance | Manager | Female | 1<br />
Finance | Manager | | 1<br />
Finance | Project Manager | Female | 1<br />
Finance | Project Manager | | 1<br />
Finance | | | 3<br />
IT | Manager | Female | 1<br />
IT | Manager | | 1<br />
IT | Project Manager | Male | 1<br />
IT | Project Manager | | 1<br />
IT | | | 2<br />
Sales | Manager | Male | 1<br />
Sales | Manager | | 1<br />
Sales | Project Manager | Male | 1<br />
Sales | Project Manager | | 1<br />
Sales | | | 2<br />
| | | 7<br />
Finance | | Female | 2<br />
IT | | Female | 1<br />
| | Female | 3<br />
Finance | | Male | 1<br />
IT | | Male | 1<br />
Sales | | Male | 2<br />
| | Male | 4<br />
| Accountant | Male | 1<br />
| Accountant | | 1<br />
| Manager | Female | 2<br />
| Manager | Male | 1<br />
| Manager | | 3<br />
| Project Manager | Female | 1<br />
| Project Manager | Male | 2<br />
| Project Manager | | 3<br />
(33 rows)<br />
<br />
So we get counts for every combination of all values. If we wanted to ensure columns are grouped in sequence, where we only summarise from the left to right, we'd use ROLLUP:<br />
<br />
# SELECT department, role, gender, count(*)<br />
FROM employees<br />
GROUP BY '''ROLLUP (department, role, gender)''';<br />
<br />
department | role | gender | count <br />
------------+-----------------+-----------+-------<br />
Finance | Accountant | Male | 1<br />
Finance | Accountant | | 1<br />
Finance | Manager | Female | 1<br />
Finance | Manager | | 1<br />
Finance | Project Manager | Female | 1<br />
Finance | Project Manager | | 1<br />
Finance | | | 3<br />
IT | Manager | Female | 1<br />
IT | Manager | | 1<br />
IT | Project Manager | Male | 1<br />
IT | Project Manager | | 1<br />
IT | | | 2<br />
Sales | Manager | Male | 1<br />
Sales | Manager | | 1<br />
Sales | Project Manager | Male | 1<br />
Sales | Project Manager | | 1<br />
Sales | | | 2<br />
| | | 7<br />
(18 rows)<br />
<br />
So we don't get summaries per role or per gender except when used in combination with the previous columns.<br />
<br />
These are just basic examples. Far more complicated configurations are possible.<br />
<br />
====Links====<br />
<br />
==JSONB-modifying operators and functions==<br />
<br />
In 9.3 (and to a greater extent in 9.4), JSONB data could be extracted using various functions and operators, but nothing that could actually modify the data. As of 9.5, JSONB data can now be modified.<br />
<br />
===jsonb || jsonb (concatenate / overwrite)===<br />
<br />
The || operator allows us to combine 2 jsonb objects. If there's overlap, values are replaced on the highest level.<br />
<br />
For example, if we want to add values to a jsonb object:<br />
<br />
# SELECT '{"name": "Joe", "age": 30}'::jsonb || '{"town": "London"}'::jsonb;<br />
?column? <br />
----------------------------------------------<br />
{"age": 30, "name": "Joe", "town": "London"}<br />
(1 row)<br />
<br />
<br />
Or we can overwrite existing values:<br />
<br />
# SELECT '{"town": "Dataville", "population": 4096}'::jsonb || '{"population": 8192}'::jsonb;<br />
?column? <br />
-------------------------------------------<br />
{"town": "Dataville", "population": 8192}<br />
(1 row)<br />
<br />
Note that this only works on the highest level, so nested objects are replaced from the top level. For example:<br />
<br />
# SELECT '{"name": "Jane", "contact": {"phone": "01234 567890", "mobile": "07890 123456"}}'::jsonb || '{"contact": {"fax": "01987 654321"}}'::jsonb;<br />
?column? <br />
------------------------------------------------------<br />
{"name": "Jane", "contact": {"fax": "01987 654321"}}<br />
(1 row)<br />
<br />
<br />
===jsonb - text / int (remove key / array element)===<br />
<br />
We can remove keys from a jsonb object with the - operator:<br />
<br />
# SELECT '{"name": "James", "email": "james@localhost"}'::jsonb - 'email';<br />
?column? <br />
-------------------<br />
{"name": "James"}<br />
(1 row)<br />
<br />
Or remove values from an array (base 0):<br />
<br />
# SELECT '["red","green","blue"]'::jsonb - 1;<br />
?column? <br />
-----------------<br />
["red", "blue"]<br />
(1 row)<br />
<br />
===jsonb - text[] / int (remove key / array element in path)===<br />
<br />
The previous example, we can remove keys or array elements, but not any lower than the highest level, so we can provide a path to the value we want to delete using a text array. Here we'll want to remove the fax number from within the contact value:<br />
<br />
# SELECT '{"name": "James", "contact": {"phone": "01234 567890", "fax": "01987 543210"}}'::jsonb - '{contact,fax}'::text[];<br />
?column? <br />
---------------------------------------------------------<br />
{"name": "James", "contact": {"phone": "01234 567890"}}<br />
(1 row)<br />
<br />
Or we can remove an array value. Here we'll get rid of the array value as index 1 (2nd value):<br />
<br />
# SELECT '{"name": "James", "aliases": ["Jamie","The Jamester","J Man"]}'::jsonb - '{aliases,1}'::text[];<br />
?column? <br />
--------------------------------------------------<br />
{"name": "James", "aliases": ["Jamie", "J Man"]}<br />
(1 row)<br />
<br />
===jsonb_replace function===<br />
<br />
The above lets us delete values in a path, but not update them, so we have the jsonb_replace function for that. We'll update the phone value within the contact value:<br />
<br />
# SELECT jsonb_replace('{"name": "James", "contact": {"phone": "01234 567890", "fax": "01987 543210"}}'::jsonb, '{contact,phone}', '"07900 112233"'::jsonb);<br />
jsonb_replace <br />
--------------------------------------------------------------------------------<br />
{"name": "James", "contact": {"fax": "01987 543210", "phone": "07900 112233"}}<br />
(1 row)<br />
<br />
===jsonb_pretty===<br />
<br />
Notice that jsonb doesn't preserve white-space, so no matter how effort you went to in order to make the object easier to read, it will end up as a long string. Well jsonb_pretty will format it for you. If we use the previous jsonb example and wrap it all in a jsonb_pretty function:<br />
<br />
# SELECT jsonb_pretty(jsonb_replace('{"name": "James", "contact": {"phone": "01234 567890", "fax": "01987 543210"}}'::jsonb, '{contact,phone}', '"07900 112233"'::jsonb));<br />
jsonb_pretty <br />
---------------------------------<br />
{ +<br />
"name": "James", +<br />
"contact": { +<br />
"fax": "01987 543210", +<br />
"phone": "07900 112233"+<br />
} +<br />
}<br />
(1 row)<br />
<br />
Much easier to read.<br />
<br />
====Links====<br />
<br />
==INSERT ... ON CONFLICT DO NOTHING/UPDATE==<br />
<br />
9.5 brings support for UPSERT operations with this additional syntax to the INSERT command. We can now tell INSERTs to switch to UPDATE operations if a conflict is found.<br />
<br />
For example, if we have a simple table with user accounts logins where we wanted to track the number of times that user had logged in:<br />
<br />
# SELECT username, logins FROM user_logins;<br />
username | logins <br />
----------+--------<br />
James | 4<br />
Lois | 2<br />
(2 rows)<br />
<br />
And we wanted to add 2 new logins, normally we'd have a problem if the primary key (or unique constraint) was violated:<br />
<br />
# INSERT INTO user_logins (username, logins)<br />
VALUES ('Naomi',1),('James',1);<br />
ERROR: duplicate key value violates unique constraint "users_pkey"<br />
DETAIL: Key (username)=(James) already exists.<br />
<br />
Unlike approaches using a Common Table Expression, the new command has no race conditions, guaranteeing either an insert or an update (provided there is no incidental error). The guarantee is maintained even in the event of many concurrent updates, inserts or deletes.<br />
<br />
Example of new syntax:<br />
<br />
# INSERT INTO user_logins (username, logins)<br />
VALUES ('Naomi',1),('James',1)<br />
'''ON CONFLICT (username)'''<br />
'''DO UPDATE SET logins = user_logins.logins + EXCLUDED.logins''';<br />
UPSERT 0 2<br />
<br />
Now let's look at what happened:<br />
<br />
# SELECT username, logins FROM user_logins;<br />
username | logins <br />
----------+--------<br />
Lois | 2<br />
Naomi | 1<br />
James | 5<br />
(3 rows)<br />
<br />
We have a new row for Naomi, which shows her having logged in once, but then we also have James whose logins value has incremented by one as specified by the UPDATE part of the statement. The UPDATE statement knows which rows it's updating based on the column or unique constraint that's being checked against.<br />
<br />
Of course there are scenarios where you might want to insert a value into a table, but only if it's not there already.<br />
<br />
Say we had a list of countries which would be used to constrain values in other tables:<br />
<br />
# SELECT * FROM countries;<br />
country <br />
-----------<br />
Australia<br />
Italy<br />
Japan<br />
UK<br />
USA<br />
(5 rows)<br />
<br />
We want to add 2 more countries. If one or more of them already existed and violated the primary key (in this case the "country" column), we'd get an error:<br />
<br />
# INSERT INTO countries (country) VALUES ('France'),('Japan');<br />
ERROR: duplicate key value violates unique constraint "countries_pkey"<br />
DETAIL: Key (country)=(Japan) already exists.<br />
<br />
But now we can tell it that a conflict is fine, and just DO NOTHING in those scenarios:<br />
<br />
# INSERT INTO countries (country) VALUES ('France'),('Japan') '''ON CONFLICT DO NOTHING''';<br />
INSERT 0 1<br />
<br />
Now we should just have one additional country in our table:<br />
<br />
# SELECT * FROM countries;<br />
country <br />
-----------<br />
Australia<br />
Italy<br />
Japan<br />
UK<br />
USA<br />
France<br />
(6 rows)<br />
<br />
If there were additional columns, also with unique constraints on, we could specify the constraint or column that we want to apply the condition to, so that a real conflict on another column produces an error.<br />
<br />
So we could have phrased that last example as:<br />
<br />
# INSERT INTO countries (country) VALUES ('France'),('Japan') '''ON CONFLICT ON CONSTRAINT countries_pkey DO NOTHING''';<br />
<br />
or<br />
<br />
# INSERT INTO countries (country) VALUES ('France'),('Japan') '''ON CONFLICT (country) DO NOTHING''';<br />
<br />
Note that providing multiple sets of conflict/update conditions isn't yet supported, so if a specific conflict is specified, but another conflict occurs instead, it will produce a conflict error like it would with a normal insert.<br />
<br />
====Links====<br />
[http://www.craigkerstiens.com/2015/05/08/upsert-lands-in-postgres-9.5/ Upsert Lands in PostgreSQL 9.5 – a First Look]<br />
<br />
[https://www.youtube.com/watch?v=pbg97bkxbbY Video - PostgreSQL 9.5's Upsert Feature Explained]<br />
<br />
==pg_rewind==<br />
<br />
pg_rewind makes it possible to efficiently bring an old primary in sync with a new primary without having to perform a full base backup. This works by looking in the Write Ahead Log to see which pages have been modified, and only copying across those pages.<br />
<br />
In this example, we have a primary (running on port 5530) and a standby subscribing to it (on port 5531):<br />
<br />
'''# SELECT * FROM pg_stat_replication;'''<br />
-[ RECORD 1 ]----+------------------------------<br />
pid | 11609<br />
usesysid | 16384<br />
usename | rep_user<br />
application_name | standby1<br />
client_addr | 127.0.0.1<br />
client_hostname | <br />
client_port | 38434<br />
backend_start | 2015-03-29 00:11:55.243319+00<br />
backend_xmin | <br />
state | streaming<br />
sent_location | 0/C81BB40<br />
write_location | 0/C81BB40<br />
flush_location | 0/C81BB40<br />
replay_location | 0/C81BB40<br />
sync_priority | 0<br />
sync_state | async<br />
<br />
Now we'll promote the standby:<br />
<br />
'''$ pg_ctl promote -D standby1'''<br />
server promoting<br />
<br />
And we'll make some changes on this instance:<br />
<br />
'''$ psql -p 5531 postgres'''<br />
<br />
'''# CREATE TABLE x (content text);'''<br />
CREATE TABLE<br />
<br />
'''# INSERT INTO x SELECT 'test' FROM generate_series(1,1000);'''<br />
INSERT 0 1000<br />
<br />
Now we'll stop old primary and use pg_rewind to re-synchronise it:<br />
<br />
'''$ pg_ctl stop -D primary'''<br />
waiting for server to shut down.... done<br />
server stopped<br />
<br />
'''$ pg_rewind -D primary --source-server='host=localhost port=5531' -P'''<br />
connected to remote server<br />
The servers diverged at WAL position 0/C81BB40 on timeline 1.<br />
Rewinding from last common checkpoint at 0/2000060 on timeline 1<br />
reading source file list<br />
reading target file list<br />
reading WAL in target<br />
Need to copy 274 MB (total source directory size is 290 MB)<br />
281142/281142 kB (100%) copied<br />
creating backup label and updating control file<br />
Done!<br />
<br />
And we'll make some changes to get it to subscribe to the new primary:<br />
<br />
'''$ cd primary'''<br />
<br />
'''$ mv recovery.{done,conf}'''<br />
<br />
'''$ vi recovery.conf''' # edited to set host info to point to port 5531 in this case<br />
<br />
'''$ vi postgresql.conf''' # as our example instances are running on the same server, we'll just change the port so it doesn't conflict<br />
<br />
Then start the new standby (old primary):<br />
<br />
'''$ pg_ctl start -D primary'''<br />
<br />
Let's see if it's successfully caught up:<br />
<br />
'''$ psql -p 5531 postgres''' # connect to the new primary<br />
'''# SELECT * FROM pg_stat_replication;'''<br />
-[ RECORD 1 ]----+------------------------------<br />
pid | 11837<br />
usesysid | 16384<br />
usename | rep_user<br />
application_name | standby1<br />
client_addr | 127.0.0.1<br />
client_hostname | <br />
client_port | 49148<br />
backend_start | 2015-03-29 00:22:39.047657+00<br />
backend_xmin | <br />
state | streaming<br />
sent_location | 0/C8559B0<br />
write_location | 0/C8559B0<br />
flush_location | 0/C8559B0<br />
replay_location | 0/C855978<br />
sync_priority | 0<br />
sync_state | async<br />
<br />
And see if the test data from the new primary is on the new standby:<br />
<br />
'''$ psql -p 5530 postgres''' # connect to the new standby<br />
<br />
'''# SELECT COUNT(*) FROM x;'''<br />
count <br />
-------<br />
1000<br />
(1 row)<br />
<br />
All synchronised.<br />
<br />
====Links====<br />
<br />
[http://www.depesz.com/2015/04/04/waiting-for-9-5-add-pg_rewind-for-re-synchronizing-a-master-server-after-failback/ Waiting for 9.5 – Add pg_rewind, for re-synchronizing a master server after failback.]<br />
<br />
=Other new features=<br />
<br />
==ALTER TABLE ... SET LOGGED / UNLOGGED==<br />
<br />
PostgreSQL allows one to create tables which aren't written to the Write Ahead Log, meaning they aren't replicated or crash-safe, but also don't have the associated overhead, so are good for data that doesn't need the guarantees of regular tables. But if you decided an unlogged table should now be replicated, or a regular table should no longer be logged, you'd previously have to create a new copy of the table and copy the data across. But in 9.5, you can switch between logged and unlogged using a new command:<br />
<br />
Set an unlogged table to logged:<br />
<br />
ALTER TABLE <tablename> SET LOGGED;<br />
<br />
Set a logged table to unlogged:<br />
<br />
ALTER TABLE <tablename> SET UNLOGGED;<br />
<br />
For example:<br />
<br />
'''# CREATE UNLOGGED TABLE messages (id int PRIMARY KEY, message text);'''<br />
<br />
'''# SELECT relname,'''<br />
'''CASE relpersistence'''<br />
'''WHEN 'u' THEN 'unlogged' '''<br />
'''WHEN 'p' then 'logged' '''<br />
'''ELSE 'unknown' END AS table_type'''<br />
'''FROM pg_class'''<br />
'''WHERE relname ~ 'messages*';'''<br />
relname | table_type <br />
---------------+------------<br />
messages | unlogged<br />
messages_pkey | unlogged<br />
(2 rows)<br />
<br />
Note that setting an unlogged table to logged will generate WAL which will contain all data in the table, so this would cause a spike in replication traffic for large tables.<br />
And now we change it to a logged table:<br />
<br />
'''# ALTER TABLE messages SET LOGGED;'''<br />
<br />
And the result of the previous query is now:<br />
<br />
relname | table_type <br />
---------------+------------<br />
messages | logged<br />
messages_pkey | logged<br />
(2 rows)<br />
<br />
==SKIP LOCKED==<br />
<br />
====Links====<br />
[http://www.depesz.com/2014/10/10/waiting-for-9-5-implement-skip-locked-for-row-level-locks/ Waiting for 9.5 – Implement SKIP LOCKED for row-level locks]<br />
<br />
==Parallel VACUUMing==<br />
<br />
The vacuumdb utility now supports parallel jobs. This is specified with the -j option, just like when using pg_dump or pg_restore. This means vacuuming a database will complete a lot quicker, and especially so for cases where tables are spread across multiple tablespaces. It will also start vacuuming the largest relations first.<br />
<br />
For example:<br />
<br />
vacuumdb -j4 productiondb<br />
<br />
This would vacuum the database named "productiondb" by spawning 4 vacuum jobs to run simultaneously.<br />
<br />
====Links====<br />
[http://www.depesz.com/2015/01/26/waiting-for-9-5-vacuumdb-enable-parallel-mode/ Waiting for 9.5 – vacuumdb: enable parallel mode]<br />
<br />
==Abbreviated Keys==<br />
<br />
The abbreviated keys optimization can be expected to greatly enhance the performance of sorts in PostgreSQL, including those used for CREATE INDEX. Reportedly, in some cases, CREATE INDEX on text columns can be as much as an entire order of magnitude faster. Numeric sorts also support the optimization.<br />
<br />
====Links====<br />
[http://pgeoghegan.blogspot.com/2015/01/abbreviated-keys-exploiting-locality-to.html Abbreviated keys: exploiting locality to improve PostgreSQL's text sort performance]<br />
<br />
[http://pgeoghegan.blogspot.com/2015/04/abbreviated-keys-for-numeric-to.html Abbreviated keys for numeric to accelerate numeric sorts]<br />
<br />
<br />
==GiST Index-Only Scans==<br />
<br />
Previously, the only index access method that supported index-only scans was B-Tree and SP-GiST, but support is added for GiST in PostgreSQL 9.5:<br />
<br />
In this example, we'll be using the btree_gist extension:<br />
<br />
'''# CREATE EXTENSION btree_gist;'''<br />
<br />
We'll set up a simple table that stores meeting room reservations:<br />
<br />
'''# CREATE TABLE meetings ('''<br />
'''id serial primary key,'''<br />
'''room int,'''<br />
'''reservation tstzrange);'''<br />
<br />
Then we add an exclusion constraint to ensure that no booking for any one room overlaps another booking for the same room, which creates an index to enforce the constraint:<br />
<br />
'''# ALTER TABLE meetings'''<br />
'''ADD CONSTRAINT meeting_exclusion'''<br />
'''EXCLUDE USING GIST (room with =, reservation with &&);'''<br />
<br />
And we'll populate it with lots of test data:<br />
<br />
'''# WITH RECURSIVE ins AS ('''<br />
'''SELECT'''<br />
'''1 AS room,'''<br />
''' '2000-01-01 08:00:00'::timestamptz AS reservation_start,'''<br />
'''(ceil(random()*24)*5 || ' minutes')::interval as duration'''<br />
'''UNION ALL'''<br />
'''SELECT'''<br />
'''CASE'''<br />
'''WHEN ins.reservation_start > now() THEN ins.room + 1'''<br />
'''ELSE ins.room'''<br />
'''END AS room,'''<br />
'''CASE'''<br />
'''WHEN ins.reservation_start > now() THEN '2000-01-01 08:00:00'::timestamptz'''<br />
'''ELSE ins.reservation_start + ins.duration'''<br />
'''END AS duration,'''<br />
'''(ceil(random()*16)*15 || ' minutes')::interval AS duration'''<br />
'''FROM ins'''<br />
'''WHERE reservation_start < now() + '1 day'::interval'''<br />
'''and room <= 200'''<br />
''')'''<br />
'''INSERT INTO meetings (room, reservation)'''<br />
'''SELECT room, tstzrange(reservation_start, reservation_start + duration) FROM ins'''<br />
'''WHERE (reservation_start + duration)::time between '08:00' and '20:00';'''<br />
<br />
One run of this results in 6.4 million rows.<br />
<br />
If we get the query plan for counting how many meetings occurred during May in 2014 for each room:<br />
<br />
'''# EXPLAIN SELECT room, count(*) FROM meetings WHERE reservation && '[2014-05-01,2014-05-31]'::tstzrange GROUP BY room ORDER BY room;'''<br />
QUERY PLAN <br />
-------------------------------------------------------------------------------------------------------------<br />
Sort (cost=1294.20..1294.21 rows=2 width=4)<br />
Sort Key: room<br />
-> HashAggregate (cost=1294.17..1294.19 rows=2 width=4)<br />
Group Key: room<br />
-> '''Index Only Scan using meeting_exclusion on meetings''' (cost=0.41..1113.98 rows=36038 width=4)<br />
Index Cond: (reservation && '["2014-05-01 00:00:00+01","2014-05-31 00:00:00+01"]'::tstzrange)<br />
<br />
<br />
Prior to 9.5, we would get the following plan:<br />
<br />
QUERY PLAN <br />
-------------------------------------------------------------------------------------------------------------------<br />
Sort (cost=28570.15..28570.16 rows=2 width=4)<br />
Sort Key: room<br />
-> HashAggregate (cost=28570.12..28570.14 rows=2 width=4)<br />
Group Key: room<br />
-> Bitmap Heap Scan on meetings (cost=778.49..28386.07 rows=36810 width=4)<br />
Recheck Cond: (reservation && '["2014-05-01 00:00:00+01","2014-05-31 00:00:00+01"]'::tstzrange)<br />
-> Bitmap Index Scan on meeting_exclusion (cost=0.00..769.29 rows=36810 width=0)<br />
Index Cond: (reservation && '["2014-05-01 00:00:00+01","2014-05-31 00:00:00+01"]'::tstzrange)<br />
<br />
== Lock scalability improvements ==<br />
<br />
The locks acquired for most operations in postgres didn't scale well to many concurrent shared acquisitions. Despite only acquiring shared locks, frequently most of the backends were blocked acquiring a exclusive spinlock. This frequently lead to 100% CPU usage by a large number of sessions. Now shared lockers should not interfere much with each others. This particularly helps on servers with more than one socket. <br />
<br />
[http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=ab5194e6f617a9a9e7aadb3dd1cee948a42d0755]<br />
<br />
== Improved buffer replacement scalability ==<br />
<br />
Before 9.5 every buffer replacement happened while holding a global lock. Now the hot path is lock-free, and the slow paths use a per buffer spinlock.<br />
<br />
[http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=5d7962c6797c0baae9ffb3b5b9ac0aec7b598bc3]<br />
[http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=d72731a70450b5e7084991b9caa15cb58a2820df]<br />
<br />
== Reduced per-backend memory usage ==<br />
<br />
Before 9.5 every backend used an array containing an entry for each page shared_buffers to manage page reference counts. With large shared_buffers that could add tens to hundreds of megabytes to each session. Now a small, fixed size, array is used instead.<br />
<br />
[http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=4b4b680c3d6d8485155d4d4bf0a92d3a874b7a65]<br />
<br />
== CustomScan interface and Join-pushdown infrastructure ==<br />
<br />
CustomScan interface allows extensions to implement own logic to scan relation, also replace built-in logic if its estimated cost is more reasonable.<br />
Relevant infrastructure also allows to replace built-in join nodes, by foreign-scan (if both child nodes are managed by same foreign server) or custom-scan; that performs like a scan on remotely/preliminary joined relations.<br />
<br />
* [http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=0b03e5951bf0a1a8868db13f02049cf686a82165 Introduce custom path and scan providers]<br />
* [http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=e7cb7ee14555cc9c5773e2c102efd6371f6f2005 Allow FDWs and custom scan providers to replace joins with scans]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGStrom&diff=25335PGStrom2015-06-28T09:03:38Z<p>Kaigai: /* Run SQL on GPU */</p>
<hr />
<div>= Overview =<br />
<br />
PG-Strom is an extension of PostgreSQL, works as custom-scan provider. It was designed to off-loads several CPU intensive workloads to GPU devices, to utilize its massive parallel execution capability.<br />
GPU has advantages on number of processor cores (usually several hundreds - thausands) and wider bandwidth of RAM (usually multiple times larger capacity than CPU). It works most effeciently when it processes massive amount of numerical operations simultaneously.<br />
<br />
PG-Strom stands on two major ideas:<br />
# Native GPU code generation on the fly<br />
# Asynchronous and pipelined execution model<br />
<br />
PG-Strom checks whether (a part of) given query is executable on GPU devices during query optimization phase. Once it determines the query is available to off-load, PG-Strom construct a source code of GPU native binary on the fly, then kicks just-in-time compile process on the head of execution stage.<br />
<br />
Next, PG-Strom loads a bunch of fetched rows on DMA buffers (15MB per buffer, in the default), then kick DMA transfer and GPU kernel execution in asynchronous manner. CUDA platform allows to handle these tasks in background, so PostgreSQL can make advance the current process.<br />
This asynchronous and in-relation sharding also hides a usual latency around GPU acceleration.<br />
<br />
[[File:PGStrom_Fig_ArchOverview.png]]<br />
<br />
<br />
= How to use =<br />
<br />
== Installation ==<br />
<br />
=== Requirements ===<br />
<br />
* PostgreSQL 9.5devel<br />
* [http://developer.nvidia.com/cuda-downloads CUDA 7.0] or later<br />
* x86_64 Linux platform supported by CUDA<br />
<br />
=== Manual Installation ===<br />
<br />
;1. Install CUDA Toolkit v7.0 (or later)<br />
<br />
* You can get CUDA Toolkit from [http://developer.nvidia.com/cuda-downloads NVIDIA], then install the toolkit. Its default installation path is <tt>/usr/local/cuda</tt>. Please don't change the default path.<br />
* For runtime library lookup, you should add a configuration to tell location of the <tt>libnvrtc.so</tt>.<br />
<br />
$ sudo bash<br />
# echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda-lib64.conf<br />
<br />
;2. Install PostgreSQL v9.5devel (or later)<br />
<br />
* You can check out PostgreSQL v9.5devel from [[http://github.com/postgres/postgres GitHUB]].<br />
* Run <tt>./configure</tt> script. At this moment, we recommend to attach <tt>--enable-debug</tt> and <tt>--enable-cassert</tt>.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/postgres/postgres.git pgsql<br />
$ cd pgsql<br />
$ ./configure --enable-debug --enable-cassert<br />
$ make<br />
$ sudo make install<br />
<br />
;3. Install latest PG-Strom<br />
<br />
* You can check out the latest PG-Strom from [[https://github.com/pg-strom/devel.git GitHUB]]<br />
* Ensure <tt>pg_config</tt> is in your command search path.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/pg-strom/devel pg_strom<br />
$ cd pg_strom<br />
$ which pg_config<br />
/usr/local/pgsql/bin/pg_config<br />
$ make<br />
$ sudo make install<br />
<br />
;4. Create datanase<br />
<br />
* Create database using <tt>initdb</tt>.<br />
* For text comparison, we recommend to add <tt>--no-local</tt> option, because collation aware text comparison is not supported on GPU side.<br />
* Edit <tt>$PGDATA/postgresql.conf</tt> for parameter configuration.<br />
** <tt>$libdir/pg_strom</tt> has to be added to <tt>shared_preload_libraries</tt>.<br />
** <tt>shared_buffers</tt> has to be expanded according to the sizing guideline.<br />
** For other configurations, see the sizing guideline.<br />
<br />
;5. Start/Restart PostgreSQL<br />
<br />
* Start PostgreSQL, using <tt>pg_ctl</tt>. You may see the following message on log.<br />
<br />
LOG: CUDA Runtime version 7.0.0<br />
LOG: NVIDIA driver version: 346.46<br />
LOG: GPU0 GeForce GTX 750 Ti (640 CUDA cores, 1110MHz), L2 2048KB, RAM 2047MB (128bits, 2700KHz), capability 5.0<br />
LOG: NVRTC - CUDA Runtime Compilation vertion 7.0<br />
LOG: database system was shut down at 2015-06-28 17:26:03 JST<br />
LOG: database system is ready to accept connections<br />
LOG: autovacuum launcher started<br />
<br />
;6. Create PG-Strom extension<br />
<br />
* Run <tt>CREATE EXTENSION pg_strom</tt> to setup relevant SQL objects.<br />
<br />
=== Cloud Installation ===<br />
<br />
Under construction<br />
<br />
== Run SQL on GPU ==<br />
<br />
Once PG-Strom is loaded, no special indication is needed to run SQL on GPU device.<br />
It perform as a custom-scan provider of PostgreSQL, and offers alternative scan/join logic that can run on GPU device. If it is feasible and reasonable from estimated cost perspective, planner put custom-scan node instead of the built-in query execution logic.<br />
Below is an ideal case to execute, because size of all the inner relations to be joined is available to load onto GPU RAM at once, and pre-aggregation can reduce number of rows to be processed by CPU effectively.<br />
<br />
postgres=# EXPLAIN SELECT cat, avg(x) FROM t0 natural join t1 natural join t2 natural join t3 natural join t4 GROUP BY cat;<br />
QUERY PLAN<br />
------------------------------------------------------------------------------------------------------------<br />
HashAggregate (cost=300706.95..300707.28 rows=26 width=12)<br />
Group Key: t0.cat<br />
-> Custom Scan (GpuPreAgg) (cost=8860.76..252963.69 rows=1077 width=52)<br />
Bulkload: On (density: 99.00%)<br />
Reduction: Local + Global<br />
-> Custom Scan (GpuJoin) (cost=7860.76..251210.47 rows=9899296 width=12)<br />
Bulkload: On (density: 100.00%)<br />
Depth 1: Logic: GpuHashJoin, HashKeys: (cid), JoinQual: (cid = cid), nrows_ratio: 0.98995197<br />
Depth 2: Logic: GpuHashJoin, HashKeys: (aid), JoinQual: (aid = aid), nrows_ratio: 1.00000000<br />
Depth 3: Logic: GpuHashJoin, HashKeys: (bid), JoinQual: (bid = bid), nrows_ratio: 1.00000000<br />
Depth 4: Logic: GpuHashJoin, HashKeys: (did), JoinQual: (did = did), nrows_ratio: 1.00000000<br />
-> Custom Scan (BulkScan) on t0 (cost=0.00..242855.74 rows=9999774 width=28)<br />
-> Seq Scan on t3 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t1 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t2 (cost=0.00..734.00 rows=40000 width=4)<br />
-> Seq Scan on t4 (cost=0.00..734.00 rows=40000 width=4)<br />
(16 rows)<br />
<br />
Below is a set of benchmark result using above microbenchmark workloads according to increase number of tables to be joined.<br />
<br />
[[File:PGStrom Fig MicroBenchGpuJoin.png]]<br />
<br />
Target Query:<br />
SELECT cat, AVG(x) FROM t0 NATURAL JOIN t1 [, ...] GROUP BY cat;<br />
<br />
;measurement environment<br />
:H/W: NEC Express5800 HR120b-1<br />
:CPU: Xeon E5-2640 (2.50GHz, 6Core)<br />
:RAM: 256GB<br />
:GPU: NVIDIA GTX980<br />
:S/W: PostgreSQL 9.5devel + PG-Strom (26-Mar-2015 ver), CUDA 7.0(x86_64)<br />
<br />
== Data types ==<br />
* Numeric<br />
** smallint<br />
** integer<br />
** bigint<br />
** real<br />
** float<br />
** numeric<br />
* Date and Time<br />
** date<br />
** time<br />
** timestamp<br />
** timestamptz<br />
* Variable length<br />
** Text<br />
** bpchar()<br />
<br />
== Functions and Operators ==<br />
<br />
Under construction<br />
<br />
== GUC options ==<br />
<br />
; pg_strom.enabled<br />
: type: boolean<br />
: default: on<br />
:It enables / disables the entire PG-Strom functionality. Once it gets disabled, CustomScan node never appear in the query execution plan.<br />
<br />
; pg_strom.perfmon<br />
: type: boolean<br />
: default: off<br />
:It enables / disables the performance monitor feature. Once this parameter gets enabled, EXPLAIN ANALYZE will show extra performance information collected from CUDA runtime.<br />
<br />
;pg_strom.cuda_visible_devices<br />
: type: text<br />
: default: NULL<br />
: It allows to configure the variable to be assigned at CUDA_VISIBLE_DEVICES environment variable on system startup time. CUDA runtime ignores the GPU devices, if this environment variable is valid but device number was not lited.<br />
: See [http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars CUDA Environment Variables] for more details.<br />
<br />
;pg_strom.enable_gpuscan<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuScan. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpuhashjoin<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuHashJoin. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpunestloop<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuNestLoop. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpupreagg<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuPreAgg. Once it got disabled, PG-Strom never inject GpuPreAgg node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpusort<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuSort. Once it got disabled, PG-Strom never inject GpuSort node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.chunk_size<br />
: type: integer<br />
: default: 15MB<br />
: The default size of individual DMA buffer (usually called chunk).<br />
<br />
; pg_strom.gpu_setup_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the average cost to set up GPU devices once per query execution. It implies the cost to initialize CUDA context and build (if not on cache) the GPU kernel source<br />
<br />
; pg_strom.gpu_operator_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each operator or function call on GPU devices<br />
<br />
; pg_strom.gpu_tuple_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each tuple (row)<br />
<br />
; pg_strom.program_cache_size<br />
: type: integer<br />
: default: 16MB<br />
: Size of CUDA program cache to be allocated on the static shared memory, during startup time once. So, too large configuration makes waste of RAM. PG-Strom caches a pair of GPU kernel source and CUDA module built, to skip run-time module build second time or later. The cached entry is kept in this area, and too small configuration makes performance degradation because of too frequent run-time compile.<br />
<br />
; pg_strom.bulkload_enabled<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the bulkload feature. If both of parent and child node of the plan tree are CustomScan node managed by PG-Strom, it checks whether bulkload mode is available to exchange the bunck of data between them.<br />
: Once PG-Strom thought bulkload is more reasonable, the child node setup its results on DMA buffer then passes this chunk to the parent node. It allows to reduce waste of CPU cycles for unnecessary data copy between normal memory area and DMA buffers. So, it is the fastest path if we can try.<br />
<br />
; pg_strom.bulkload_density<br />
: type: float<br />
: default: 0.50<br />
: It sets a threshold to use bulkloading mode to exchange a bunch of data from the underlying node of the plan tree. If child node may increase / reduce number of tuples more / less than the threshold, we don't apply bulkload mode even if possible.<br />
<br />
; pg_strom.max_async_tasks<br />
: type: integer<br />
: default: 32<br />
: It sets maximum number of asynchronous tasks that are sum of launched and pending. When GPU kernel execution or GPU kernel build (usual case) takes times, PG-Strom makes advance child node scanning and loading to DMA buffer. This parameter works as threshold of this prefetching.<br />
<br />
; pg_strom.max_workers<br />
: type: integer<br />
: default: ---<br />
: It sets maximum number of background worker process if and when PG-Strom also takes CPU parallel execution.<br />
<br />
== Sizing Guideline ==<br />
<br />
Under construction<br />
<br />
<br />
= Background =<br />
<br />
== Parallel Approach ==<br />
<br />
We have a few ways to enhance performance of PostgreSQL.<br />
* Homogeneous scale-up<br />
* Heterogeneous scale-up<br />
* Scale-out<br />
PG-Strom takes heterogeneous scale-up approach; that utilizes the most advantaged hardware according to the characteristics of the workloads. In other words, it assigns simple but massive amount of numerical calculation, that is too previous to run on CPU cores, on GPU devices.<br />
<br />
[[File:PGStrom_Fig_ParallelApproach.png]]<br />
<br />
= Future Development =<br />
<br />
== PostgreSQL core ==<br />
<br />
== PG-Strom ==<br />
<br />
== Long term development ==</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PGStrom&diff=25334PGStrom2015-06-28T08:28:35Z<p>Kaigai: </p>
<hr />
<div>= Overview =<br />
<br />
PG-Strom is an extension of PostgreSQL, works as custom-scan provider. It was designed to off-loads several CPU intensive workloads to GPU devices, to utilize its massive parallel execution capability.<br />
GPU has advantages on number of processor cores (usually several hundreds - thausands) and wider bandwidth of RAM (usually multiple times larger capacity than CPU). It works most effeciently when it processes massive amount of numerical operations simultaneously.<br />
<br />
PG-Strom stands on two major ideas:<br />
# Native GPU code generation on the fly<br />
# Asynchronous and pipelined execution model<br />
<br />
PG-Strom checks whether (a part of) given query is executable on GPU devices during query optimization phase. Once it determines the query is available to off-load, PG-Strom construct a source code of GPU native binary on the fly, then kicks just-in-time compile process on the head of execution stage.<br />
<br />
Next, PG-Strom loads a bunch of fetched rows on DMA buffers (15MB per buffer, in the default), then kick DMA transfer and GPU kernel execution in asynchronous manner. CUDA platform allows to handle these tasks in background, so PostgreSQL can make advance the current process.<br />
This asynchronous and in-relation sharding also hides a usual latency around GPU acceleration.<br />
<br />
[[File:PGStrom_Fig_ArchOverview.png]]<br />
<br />
<br />
= How to use =<br />
<br />
== Installation ==<br />
<br />
=== Requirements ===<br />
<br />
* PostgreSQL 9.5devel<br />
* [http://developer.nvidia.com/cuda-downloads CUDA 7.0] or later<br />
* x86_64 Linux platform supported by CUDA<br />
<br />
=== Manual Installation ===<br />
<br />
;1. Install CUDA Toolkit v7.0 (or later)<br />
<br />
* You can get CUDA Toolkit from [http://developer.nvidia.com/cuda-downloads NVIDIA], then install the toolkit. Its default installation path is <tt>/usr/local/cuda</tt>. Please don't change the default path.<br />
* For runtime library lookup, you should add a configuration to tell location of the <tt>libnvrtc.so</tt>.<br />
<br />
$ sudo bash<br />
# echo /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda-lib64.conf<br />
<br />
;2. Install PostgreSQL v9.5devel (or later)<br />
<br />
* You can check out PostgreSQL v9.5devel from [[http://github.com/postgres/postgres GitHUB]].<br />
* Run <tt>./configure</tt> script. At this moment, we recommend to attach <tt>--enable-debug</tt> and <tt>--enable-cassert</tt>.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/postgres/postgres.git pgsql<br />
$ cd pgsql<br />
$ ./configure --enable-debug --enable-cassert<br />
$ make<br />
$ sudo make install<br />
<br />
;3. Install latest PG-Strom<br />
<br />
* You can check out the latest PG-Strom from [[https://github.com/pg-strom/devel.git GitHUB]]<br />
* Ensure <tt>pg_config</tt> is in your command search path.<br />
* Run <tt>make</tt> and <tt>make install</tt><br />
<br />
$ git clone https://github.com/pg-strom/devel pg_strom<br />
$ cd pg_strom<br />
$ which pg_config<br />
/usr/local/pgsql/bin/pg_config<br />
$ make<br />
$ sudo make install<br />
<br />
;4. Create datanase<br />
<br />
* Create database using <tt>initdb</tt>.<br />
* For text comparison, we recommend to add <tt>--no-local</tt> option, because collation aware text comparison is not supported on GPU side.<br />
* Edit <tt>$PGDATA/postgresql.conf</tt> for parameter configuration.<br />
** <tt>$libdir/pg_strom</tt> has to be added to <tt>shared_preload_libraries</tt>.<br />
** <tt>shared_buffers</tt> has to be expanded according to the sizing guideline.<br />
** For other configurations, see the sizing guideline.<br />
<br />
;5. Start/Restart PostgreSQL<br />
<br />
* Start PostgreSQL, using <tt>pg_ctl</tt>. You may see the following message on log.<br />
<br />
LOG: CUDA Runtime version 7.0.0<br />
LOG: NVIDIA driver version: 346.46<br />
LOG: GPU0 GeForce GTX 750 Ti (640 CUDA cores, 1110MHz), L2 2048KB, RAM 2047MB (128bits, 2700KHz), capability 5.0<br />
LOG: NVRTC - CUDA Runtime Compilation vertion 7.0<br />
LOG: database system was shut down at 2015-06-28 17:26:03 JST<br />
LOG: database system is ready to accept connections<br />
LOG: autovacuum launcher started<br />
<br />
;6. Create PG-Strom extension<br />
<br />
* Run <tt>CREATE EXTENSION pg_strom</tt> to setup relevant SQL objects.<br />
<br />
=== Cloud Installation ===<br />
<br />
Under construction<br />
<br />
== Run SQL on GPU ==<br />
<br />
<br />
== Data types ==<br />
* Numeric<br />
** smallint<br />
** integer<br />
** bigint<br />
** real<br />
** float<br />
** numeric<br />
* Date and Time<br />
** date<br />
** time<br />
** timestamp<br />
** timestamptz<br />
* Variable length<br />
** Text<br />
** bpchar()<br />
<br />
== Functions and Operators ==<br />
<br />
Under construction<br />
<br />
== GUC options ==<br />
<br />
; pg_strom.enabled<br />
: type: boolean<br />
: default: on<br />
:It enables / disables the entire PG-Strom functionality. Once it gets disabled, CustomScan node never appear in the query execution plan.<br />
<br />
; pg_strom.perfmon<br />
: type: boolean<br />
: default: off<br />
:It enables / disables the performance monitor feature. Once this parameter gets enabled, EXPLAIN ANALYZE will show extra performance information collected from CUDA runtime.<br />
<br />
;pg_strom.cuda_visible_devices<br />
: type: text<br />
: default: NULL<br />
: It allows to configure the variable to be assigned at CUDA_VISIBLE_DEVICES environment variable on system startup time. CUDA runtime ignores the GPU devices, if this environment variable is valid but device number was not lited.<br />
: See [http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars CUDA Environment Variables] for more details.<br />
<br />
;pg_strom.enable_gpuscan<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuScan. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpuhashjoin<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuHashJoin. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpunestloop<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuNestLoop. Once it got disabled, PG-Strom never uses GpuScan during plan construction. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpupreagg<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuPreAgg. Once it got disabled, PG-Strom never inject GpuPreAgg node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.enable_gpusort<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the feature support of GpuSort. Once it got disabled, PG-Strom never inject GpuSort node to the PlannedStmt. Note that it does not affect to behavior of the query execution plan already constructed.<br />
<br />
; pg_strom.chunk_size<br />
: type: integer<br />
: default: 15MB<br />
: The default size of individual DMA buffer (usually called chunk).<br />
<br />
; pg_strom.gpu_setup_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the average cost to set up GPU devices once per query execution. It implies the cost to initialize CUDA context and build (if not on cache) the GPU kernel source<br />
<br />
; pg_strom.gpu_operator_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each operator or function call on GPU devices<br />
<br />
; pg_strom.gpu_tuple_cost<br />
: type: float<br />
: default: to be determined based on the device property<br />
: The planner's estimate of the cost of processing each tuple (row)<br />
<br />
; pg_strom.program_cache_size<br />
: type: integer<br />
: default: 16MB<br />
: Size of CUDA program cache to be allocated on the static shared memory, during startup time once. So, too large configuration makes waste of RAM. PG-Strom caches a pair of GPU kernel source and CUDA module built, to skip run-time module build second time or later. The cached entry is kept in this area, and too small configuration makes performance degradation because of too frequent run-time compile.<br />
<br />
; pg_strom.bulkload_enabled<br />
: type: boolean<br />
: default: on<br />
: It enables / disables the bulkload feature. If both of parent and child node of the plan tree are CustomScan node managed by PG-Strom, it checks whether bulkload mode is available to exchange the bunck of data between them.<br />
: Once PG-Strom thought bulkload is more reasonable, the child node setup its results on DMA buffer then passes this chunk to the parent node. It allows to reduce waste of CPU cycles for unnecessary data copy between normal memory area and DMA buffers. So, it is the fastest path if we can try.<br />
<br />
; pg_strom.bulkload_density<br />
: type: float<br />
: default: 0.50<br />
: It sets a threshold to use bulkloading mode to exchange a bunch of data from the underlying node of the plan tree. If child node may increase / reduce number of tuples more / less than the threshold, we don't apply bulkload mode even if possible.<br />
<br />
; pg_strom.max_async_tasks<br />
: type: integer<br />
: default: 32<br />
: It sets maximum number of asynchronous tasks that are sum of launched and pending. When GPU kernel execution or GPU kernel build (usual case) takes times, PG-Strom makes advance child node scanning and loading to DMA buffer. This parameter works as threshold of this prefetching.<br />
<br />
; pg_strom.max_workers<br />
: type: integer<br />
: default: ---<br />
: It sets maximum number of background worker process if and when PG-Strom also takes CPU parallel execution.<br />
<br />
== Sizing Guideline ==<br />
<br />
Under construction<br />
<br />
<br />
= Background =<br />
<br />
== Parallel Approach ==<br />
<br />
We have a few ways to enhance performance of PostgreSQL.<br />
* Homogeneous scale-up<br />
* Heterogeneous scale-up<br />
* Scale-out<br />
PG-Strom takes heterogeneous scale-up approach; that utilizes the most advantaged hardware according to the characteristics of the workloads. In other words, it assigns simple but massive amount of numerical calculation, that is too previous to run on CPU cores, on GPU devices.<br />
<br />
[[File:PGStrom_Fig_ParallelApproach.png]]<br />
<br />
= Future Development =<br />
<br />
== PostgreSQL core ==<br />
<br />
== PG-Strom ==<br />
<br />
== Long term development ==</div>Kaigaihttps://wiki.postgresql.org/index.php?title=File:PGStrom_Fig_ParallelApproach.png&diff=25333File:PGStrom Fig ParallelApproach.png2015-06-28T05:31:32Z<p>Kaigai: </p>
<hr />
<div></div>Kaigaihttps://wiki.postgresql.org/index.php?title=File:PGStrom_Fig_ArchOverview.png&diff=25332File:PGStrom Fig ArchOverview.png2015-06-28T05:30:56Z<p>Kaigai: </p>
<hr />
<div></div>Kaigaihttps://wiki.postgresql.org/index.php?title=File:PGStrom_Fig_MicroBenchGpuJoin.png&diff=25331File:PGStrom Fig MicroBenchGpuJoin.png2015-06-28T05:30:24Z<p>Kaigai: </p>
<hr />
<div></div>Kaigaihttps://wiki.postgresql.org/index.php?title=File:PGStrom_Fig_GpuScan.png&diff=25330File:PGStrom Fig GpuScan.png2015-06-28T05:29:42Z<p>Kaigai: </p>
<hr />
<div></div>Kaigaihttps://wiki.postgresql.org/index.php?title=File:PGStrom_Fig_GpuNestLoop.png&diff=25329File:PGStrom Fig GpuNestLoop.png2015-06-28T05:28:25Z<p>Kaigai: </p>
<hr />
<div></div>Kaigaihttps://wiki.postgresql.org/index.php?title=File:PGStrom_Fig_Gpu_Reduction.png&diff=25328File:PGStrom Fig Gpu Reduction.png2015-06-28T05:27:57Z<p>Kaigai: </p>
<hr />
<div>== Licensing ==<br />
{{The PostgreSQL Licence}}</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=25194PgCon 2015 Developer Unconference2015-06-17T16:22:45Z<p>Kaigai: /* Meeting Notes */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
== Schedule ==<br />
<br />
[[PgCon_2015_Developer_Unconference_Schedule|Schedule is here]]. It is updated frequently.<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
# Ingmar Alting<br />
# Naoya Anzai (arrive tuesday evening)<br />
# Chris Autry<br />
# Ashutosh Bapat<br />
# Oleg Bartunov<br />
# Josh Berkus<br />
# Christopher Browne<br />
# Joe Conway<br />
# Jeff Davis<br />
# Andrew Dunstan<br />
# Yurie Enomoto<br />
# Ozgun Erdogan (Wednesday)<br />
# Ed Espino<br />
# Andres Freund<br />
# Stephen Frost<br />
# Masao Fujii<br />
# Etsuro Fujita<br />
# Peter Geoghegan<br />
# Kevin Grittner<br />
# Robert Haas<br />
# Ahsan Hadi<br />
# Magnus Hagander<br />
# Shigeru Hanada<br />
# Álvaro Herrera<br />
# Yasuo Honda<br />
# Kyotaro Horiguchi<br />
# Thierry Husson (Wednesday @ 11am)<br />
# Ayumi Ishii<br />
# Tatsuo Ishii<br />
# Moshe Jacobson<br />
# Stefan Kaltenbrunner<br />
# Amit Kapila<br />
# Mehmet Emin KARAKAŞ<br />
# Motoyuki Kawaba (arrive Tuesday evening)<br />
# Konstantin Knizhnik<br />
# KaiGai Kohei (arrive tuesday evening)<br />
# Alexander Korotkov<br />
# Ilya Kosmodemiansky<br />
# Dilip Kumar<br />
# Tom Lane<br />
# Amit Langote<br />
# Heikki Linnakangas<br />
# Chris Malek (Wednesday @ 11am)<br />
# Grant McAlister<br />
# Mack McCauley<br />
# Fabrízio de Royes Mello<br />
# Noah Misch<br />
# Bruce Momjian<br />
# Yugo Nagata<br />
# Satoshi Nagayasu<br />
# Jim Nasby<br />
# Dave Page<br />
# Christophe Pettus<br />
# Tyler Poland<br />
# Paul Ramsey<br />
# Kumar Rajeev Rastogi<br />
# Simon Riggs<br />
# Michael Robinson<br />
# Tetsuo Sakata<br />
# Masahiko Sawada<br />
# Arul Shaji<br />
# Dan Shuster<br />
# Steve Singer (arrive tuesday mid-afternoon)<br />
# Marco Slot <br />
# Greg Smith<br />
# David Steele (arrive tuesday evening)<br />
# Jose Luis Tallon (arrives tuesday evening)<br />
# Yasin TATAR<br />
# Euler Taveira<br />
# Rod Taylor<br />
# Fabio Telles<br />
# Jan Urbański (Wednesday)<br />
# Tomas Vondra<br />
# Jan Wieck (arrive tuesday evening)<br />
# Chris Winters<br />
# Nat Wyatt<br />
# Jimmy Yih<br />
# Rob Young<br />
<br />
=Topics=<br />
<br />
''See the schedule at [[PgCon 2015 Developer Unconference Schedule]]''<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|Picture!<br />
|Open<br />
|Oleg Bartunov (Fuji 16-55)<br />
|All!<br />
|All!<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander, Joe Conway, David Steele, Fabrízio de Royes Mello, Satoshi Nagayasu, Dave Cramer, Alexander Korotkov, Chris Malek, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus, Joe Conway, Michael Robinson, Oleg Bartunov, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|Amit Kapila<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen, Ilya Kosmodemiansky, Tomas Vondra, Grant McAlister, Joe Conway, Peter Geoghegan, Kyotaro Horiguchi, Simon Riggs, Amit Langote, Andres Freund, Robert Haas, David Steele, Rod Taylor, Jim Nasby, Chris Winters, Nat Wyatt, Noah Misch, Masao Fujii, Mehmet Emin KARAKAŞ, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Andrew Dunstan, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Dave Cramer, Steve Singer, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Marc Jeanneret, Chris Malek<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|Noah Misch, Álvaro Herrera, Andres Freund, Robert Haas, Tom Lane, Andrew Dunstan,Oleg Bartunov<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis, Ozgun Erdogan, Tomas Vondra, Peter Geoghegan, Robert Haas, Chris Browne, Josh Berkus, Ingmar Alting, Masao Fujii, Christophe Pettus, Jose Luis Tallon, Heikki Linnakangas, Alexander Korotkov,Oleg Bartunov,Konstantin Knizhnik<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Tetsuo Sakata, Simon Riggs, Robert Haas, David Steele, Rod Taylor, Chris Browne, Jim Nasby, Josh Berkus, Chris Winters, Masao Fujii, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Satoshi Nagayasu, Andrew Dunstan, Mack McCauley, Shigeru HANADA, Michael Robinson, Steve Singer, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Andres Freund, Marc Jeanneret, Chris Malek, Marco Slot<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Closed<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|During Lunch Wed.<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat, Ahsan Hadi, Yurie Enomoto, Chris Malek<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Kyotaro Horiguchi, KaiGai Kohei, Noah Misch, Tetsuo Sakata, Peter Geoghegan, Álvaro Herrera, Thierry Husson, Joe Conway, Naoya Anzai, Robert Haas, David Steele, Chris Browne, Jim Nasby, Josh Berkus, Masao Fujii, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Andrew Dunstan, Jose Luis Tallon, Yurie Enomoto, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Yasuo Honda, Dave Cramer,Steve Singer, Alexander Korotkov, Oleg Bartunov,Konstantin Knizhnik, Chris Malek, Ed Espino, Keith Fiske<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Noah Misch, Tetsuo Sakata, Naoya Anzai, Robert Haas, Jim Nasby, Josh Berkus, Chris Winters, Ingmar Alting, Mehmet Emin KARAKAŞ, Jose Luis Tallon, Oleg Bartunov, Konstanti Knizhnik, Chris Malek, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductors - GPU, SSD, NVRAM, FPGA, PMEM...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|Matthew Wilcox, Josh Berkus, Satoshi Nagayasu, Jose Luis Tallon, Naoya Anzai, Mack McCauley, Shigeru HANADA, Michael Robinson, Ingmar Alting<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Columnar Storage<br />
|Open<br />
|<br />
|Álvaro Herrera<br />
|Ozgun Erdogan, Tomas Vondra, KaiGai Kohei, Amit Kapila, Josh Berkus, Naoya Anzai, Amit Langote, Robert Haas, David Steele, Rod Taylor, Chris Browne, Jim Nasby, Chris Winters, Nat Wyatt, Masao Fujii, Fabrízio de Royes Mello, Euler Taveira, Satoshi Nagayasu, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Heikki Linnakangas, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Andres Freund, Marc Jeanneret, Marco Slot, Ed Espino, Arul Shaji, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis, Amit Langote, Kumar Rajeev Rastogi, Josh Berkus, Simon Riggs, Robert Haas, Jim Nasby, Masao Fujii, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Yurie Enomoto, Masahiko Sawada, Shigeru HANADA, Yasuo Honda,Steve Singer, Marc Jeanneret<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|Kumar Rajeev Rastogi, Noah Misch, Ilya Kosmodemiansky, Amit Kapila, Simon Riggs, Rod Taylor, Jim Nasby, Josh Berkus, Nat Wyatt, Christophe Pettus, Satoshi Nagayasu<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Slony Development]]<br />
|Open<br />
|<br />
| Steve Singer, Chris Browne, Jan Wieck<br />
| Josh Berkus, Rod Taylor, Jim Nasby, Satoshi Nagayasu, Yurie Enomoto<br />
<br />
|- style="background-color:lightgray;"<br />
|[[DockerizingPostgres|Dockerizing Postgres]]<br />
|Open<br />
|<br />
| Josh Berkus<br />
| Simon Riggs, Nat Wyatt, Christophe Pettus, Fabrízio de Royes Mello, Jan Urbański, Michael Robinson, Jeff Davis, Rob Young, Greg Smith, Keith Fiske<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Bi Directional Replication & Logical Decoding|BDR]]<br />
|Open<br />
|<br />
| Simon Riggs<br />
| Andres Freund, Jim Nasby, Josh Berkus, Mehmet Emin KARAKAŞ, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Michael Robinson, Dave Cramer,Steve Singer, Jeff Davis, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|[[AutonomousTransactionsUnconference2015|Autonomous Transactions]]<br />
|Open<br />
|<br />
| Simon Riggs, Kumar Rajeev Rastogi<br />
| David Steele, Jim Nasby, Josh Berkus, Nat Wyatt, Masao Fujii, Euler Taveira, Andrew Dunstan, Masahiko Sawada, Michael Robinson, Amit Kapila, Chris Malek, Arul Shaji, Fabrízio de Royes Mello<br />
<br />
|- style="background-color:lightgray;"<br />
|Audit Logging<br />
|Open<br />
|<br />
| David Steele<br />
| Josh Berkus, Nat Wyatt, Masao Fujii, Christophe Pettus, Fabio Telles, Satoshi Nagayasu, Yurie Enomoto, Mack McCauley, Masahiko Sawada, Michael Robinson, Oleg Bartunov, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pg_shard v2.0 and Lessons Learned from NoSQL Databases ]]<br />
|Open<br />
|<br />
| Ozgun Erdogan, Marco Slot <br />
| Josh Berkus, Jim Nasby, Josh Berkus, Chris Winters, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Satoshi Nagayasu, Shigeru HANADA, Michael Robinson, Oleg Bartunov, Chris Malek<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Direction of json and jsonb<br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Christophe Pettus, Masahiko Sawada, Michael Robinson, Rod Taylor, Alexander Korotkov, Oleg Bartunov, Chris Malek, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Sparse Set Type <br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Michael Robinson<br />
<br />
|- style="background-color:lightgray;"<br />
|Testing Framework Adequacy<br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Christophe Pettus, Mack McCauley, Michael Robinson,Steve Singer, Oleg Bartunov, Ed Espino<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Vertical Scalability w.r.t Writes ==<br />
Purpose of this discussion:<br />
* Discuss about priority/importance of various performance and scalability problems<br />
* Solution/Idea to solve most important problem('s)<br />
* Is pgbench sufficient to capture various kind of real world workloads?<br />
<br />
Some of important performance problems I have in mind are:<br />
<br />
* Avoid/Reduce Vacuum Freeze<br />
* Bloat<br />
** Heap<br />
** Index<br />
* Instability in TPS due to checkpointer flush<br />
* Tuple size<br />
** Heap Tuple Header <br />
** Alignment in index can lead to bigger index size for simple datatypes<br />
<br />
'''Scalability bottlenecks'''<br />
* Locks<br />
** ProcArrayLock<br />
** WALWriteLock<br />
** CLOGControlLock<br />
** Lock for Relation Extension<br />
<br />
* Writes, especially when data doesn't fit in shared buffers.<br />
** Write Performance<br />
** Double Buffering<br />
** In-memory table/tablespaces<br />
<br />
=== Meeting Notes ===<br />
<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* This will be, ehem, secure so nothing will be written here<br />
<br />
== Partitioning ==<br />
Proposal to enhance partitioning support in PostgreSQL was posted to -hackers last year and resulted in discussion of some ideas regarding implementation. Late in the discussion, a crude WIP patch was also posted with some experimental syntax, catalog changes, an idea for internal representation and a proof-of-concept INSERT tuple routing function demonstrating practicality of the internal representation. It would be nice to carry the discussion forward at the same time implementing a patch to be proposed, reviewed early in the 9.6 development cycle. Points to discuss could be: <br />
<br />
* New features and old inheritance based implementation<br />
* Planner considerations for new partitioned table<br />
* Need for a new Append-like executor node for partitioned tables<br />
* DML/DDL restrictions on partitioned tables and partitions<br />
* Basically any considerations for partitioned tables and partitions that are explicitly defined so at a layer that's above the storage layer<br />
* Other points that come up<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductors ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* Differences in APIs between PMEM and SSD/Disk - Unlike traditional device, PMEM assumes memory mapped cacheline based interface. PostgreSQL's storage manager is a good candidate for investigation to support both type of devices; one is traditional VFS based one, other is memory mapped one. PMEM library will offer clear interface to applications.<br />
* If all system RAM is persistent, we don't need transaction log, however, we are in the meantime of technology improvement. In the near future, we utilize PMEM on a particular portion, like transaction log buffer.<br />
* Does GPU help data encryption/description? Yes, it may be possible, probably, special code shall implement type input/output handler. However, a certain level of complexity of calculation is needed to make sense GPU acceleration.<br />
* In PG-Strom case, GPU works well to the type of workload to reduce number of rows; scan, join (with equal condition) or aggregation.<br />
* PL function is predefined, so it may work fine even if HDL build and device rewrite takes 30minutes.<br />
* XeonPhi needs another optimization than GPU. Because GPU logic usually has inter-thread-synchronization, however, it is emulated by software on XeonPhi. It is too challenging to write once, run anywhere,<br />
* GPU and AVX (SIMD operation) will work ideally on column oriented data structure, so Alvaro's work is very desirable from the standpoint of HW acceleration.<br />
<br />
* [https://wiki.postgresql.org/images/6/66/20150617_PGconf2015_Unconf_KaiGai.pdf discussion material]<br />
* [http://pmem.io/nvml/ NVM Library]<br />
<br />
* Action items<br />
** investigation of storage manager API to fit both of VFS and PMEM<br />
** columnar storage as better infrastructure of HW acceleration<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Native Columnar Storage ==<br />
<br />
See Alvaro's [http://www.postgresql.org/message-id/20150611230316.GM133018@postgresql.org email to Hackers].<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Audit Logging ==<br />
<br />
Audit logging is an important part of a RDBMS for many users and applications. Discuss how best to incorporate audit logging into PostgreSQL and what must be included at a minimum to make the feature viable. <br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Direction of json and jsonb ==<br />
<br />
What are the future needs of the JSON types? Recent suggestions have included an indexable "exists" operator, the JSON pointer and JSON patch standards, <br />
recursive merge, intersection, and being able to assign to a subdocument (json#>path as an lvalue). What are people using these types for, and what are <br />
the major gaps in functionality?<br />
<br />
=== Meeting Notes ===<br />
<br />
* Tom: need to work on selectivity estimators for the indexable operators<br />
* Oleg: update syntax wanted<br />
** UPDATE table SET jsoncol['a'][1]['b'] = ...jsonvalue...<br />
** similar syntax for fetching a value<br />
** potentially, most container types should be able to offer array-like fetch and set syntax<br />
* Oleg: jsquery-like query language<br />
** there will be a talk about that tomorrow<br />
** possible patch in 9.7 timeframe<br />
* Oleg: validation of JSON values against a schema<br />
** is there any accepted standard for JSON schemas?<br />
<br />
=== Attendees ===<br />
* Tom Lane<br />
<br />
== Native Sparse Set Type ==<br />
<br />
Sets over small domains can be reasonably modeled by bitmaps, but sets over very large domains can not.<br />
Is there a need for such sets? How would we implement them? Arrays? Balanced trees? Something else?<br />
What types of sets would we allow? Anything with Btree operators, or more restricted? What would the notation look like?<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Testing Framework Adequacy ==<br />
<br />
The buildfarm is more than 10 years old, and the testing needs of Postgres and its ofware ecosystem have changed radically in that time.<br />
What do we now need in the way of testing? How do we test complex arrangements such as the various sorts of replication in an automated way?<br />
Do we need a new framwork, or can the existing framework be adapted to our needs?<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
<br />
== Unconference schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!DMS1110<br />
!DMS1120<br />
!DMS1160<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 13:30-14:30<br />
|<br />
|PostgreSQL and SMR Drives - the future of magnetic storage means very expensive random writes<br />
|pgPool2 towards version 3.5<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 14:45-15:45<br />
|Autonomous Transactions<br />
|Native Compilation + LLVM<br />
|pgAdmin4<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 16:00-17:30<br />
|Bi Directional Replication & Logical Decoding&#124;BDR<br />
|<br />
|Advocacy Team Meeting<br />
<br />
|- style="background-color:lightgray;"<br />
|<br />
|<br />
|<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 10:00-11:15<br />
|Security Team Meeting<br />
|Slony Development<br />
|Utilization of modern semiconductors - GPU, SSD, NVRAM, FPGA, PMEM...<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 11:30-13:00<br />
|Direction of json and jsonb<br />
|Dockerizing Postgres<br />
|Vertical Scalability w.r.t Write<br />
<br />
|- style="background-color:lightgray;"<br />
|Lunch time<br />
|PGCAC Board Meeting 2015<br />
|<br />
|LUNCH!!<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 14:00-15:00<br />
|Native Sparse Set Type<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Horizontal Scalability / Sharding in PostgreSQL - ground covered so far and remaining to be covered.<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 15:15-16:15<br />
|pg_shard v2.0 and Lessons Learned from NoSQL Databases<br />
|Testing Framework Adequacy<br />
|Native Columnar Storage<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 16:30-17:30<br />
|Foreign Data Wrapper enhancements<br />
|Audit Logging<br />
|Partitioning<br />
<br />
|- style="background-color:lightgray;"<br />
|17:30<br />
|<br />
|<br />
|Picture!</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=25192PgCon 2015 Developer Unconference2015-06-17T16:06:01Z<p>Kaigai: /* Meeting Notes */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
== Schedule ==<br />
<br />
[[PgCon_2015_Developer_Unconference_Schedule|Schedule is here]]. It is updated frequently.<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
# Ingmar Alting<br />
# Naoya Anzai (arrive tuesday evening)<br />
# Chris Autry<br />
# Ashutosh Bapat<br />
# Oleg Bartunov<br />
# Josh Berkus<br />
# Christopher Browne<br />
# Joe Conway<br />
# Jeff Davis<br />
# Andrew Dunstan<br />
# Yurie Enomoto<br />
# Ozgun Erdogan (Wednesday)<br />
# Ed Espino<br />
# Andres Freund<br />
# Stephen Frost<br />
# Masao Fujii<br />
# Etsuro Fujita<br />
# Peter Geoghegan<br />
# Kevin Grittner<br />
# Robert Haas<br />
# Ahsan Hadi<br />
# Magnus Hagander<br />
# Shigeru Hanada<br />
# Álvaro Herrera<br />
# Yasuo Honda<br />
# Kyotaro Horiguchi<br />
# Thierry Husson (Wednesday @ 11am)<br />
# Ayumi Ishii<br />
# Tatsuo Ishii<br />
# Moshe Jacobson<br />
# Stefan Kaltenbrunner<br />
# Amit Kapila<br />
# Mehmet Emin KARAKAŞ<br />
# Motoyuki Kawaba (arrive Tuesday evening)<br />
# Konstantin Knizhnik<br />
# KaiGai Kohei (arrive tuesday evening)<br />
# Alexander Korotkov<br />
# Ilya Kosmodemiansky<br />
# Dilip Kumar<br />
# Tom Lane<br />
# Amit Langote<br />
# Heikki Linnakangas<br />
# Chris Malek (Wednesday @ 11am)<br />
# Grant McAlister<br />
# Mack McCauley<br />
# Fabrízio de Royes Mello<br />
# Noah Misch<br />
# Bruce Momjian<br />
# Yugo Nagata<br />
# Satoshi Nagayasu<br />
# Jim Nasby<br />
# Dave Page<br />
# Christophe Pettus<br />
# Tyler Poland<br />
# Paul Ramsey<br />
# Kumar Rajeev Rastogi<br />
# Simon Riggs<br />
# Michael Robinson<br />
# Tetsuo Sakata<br />
# Masahiko Sawada<br />
# Arul Shaji<br />
# Dan Shuster<br />
# Steve Singer (arrive tuesday mid-afternoon)<br />
# Marco Slot <br />
# Greg Smith<br />
# David Steele (arrive tuesday evening)<br />
# Jose Luis Tallon (arrives tuesday evening)<br />
# Yasin TATAR<br />
# Euler Taveira<br />
# Rod Taylor<br />
# Fabio Telles<br />
# Jan Urbański (Wednesday)<br />
# Tomas Vondra<br />
# Jan Wieck (arrive tuesday evening)<br />
# Chris Winters<br />
# Nat Wyatt<br />
# Jimmy Yih<br />
# Rob Young<br />
<br />
=Topics=<br />
<br />
''See the schedule at [[PgCon 2015 Developer Unconference Schedule]]''<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|Picture!<br />
|Open<br />
|Oleg Bartunov (Fuji 16-55)<br />
|All!<br />
|All!<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander, Joe Conway, David Steele, Fabrízio de Royes Mello, Satoshi Nagayasu, Dave Cramer, Alexander Korotkov, Chris Malek, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus, Joe Conway, Michael Robinson, Oleg Bartunov, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|Amit Kapila<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen, Ilya Kosmodemiansky, Tomas Vondra, Grant McAlister, Joe Conway, Peter Geoghegan, Kyotaro Horiguchi, Simon Riggs, Amit Langote, Andres Freund, Robert Haas, David Steele, Rod Taylor, Jim Nasby, Chris Winters, Nat Wyatt, Noah Misch, Masao Fujii, Mehmet Emin KARAKAŞ, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Andrew Dunstan, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Dave Cramer, Steve Singer, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Marc Jeanneret, Chris Malek<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|Noah Misch, Álvaro Herrera, Andres Freund, Robert Haas, Tom Lane, Andrew Dunstan,Oleg Bartunov<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis, Ozgun Erdogan, Tomas Vondra, Peter Geoghegan, Robert Haas, Chris Browne, Josh Berkus, Ingmar Alting, Masao Fujii, Christophe Pettus, Jose Luis Tallon, Heikki Linnakangas, Alexander Korotkov,Oleg Bartunov,Konstantin Knizhnik<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Tetsuo Sakata, Simon Riggs, Robert Haas, David Steele, Rod Taylor, Chris Browne, Jim Nasby, Josh Berkus, Chris Winters, Masao Fujii, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Satoshi Nagayasu, Andrew Dunstan, Mack McCauley, Shigeru HANADA, Michael Robinson, Steve Singer, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Andres Freund, Marc Jeanneret, Chris Malek, Marco Slot<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Closed<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|During Lunch Wed.<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat, Ahsan Hadi, Yurie Enomoto, Chris Malek<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Kyotaro Horiguchi, KaiGai Kohei, Noah Misch, Tetsuo Sakata, Peter Geoghegan, Álvaro Herrera, Thierry Husson, Joe Conway, Naoya Anzai, Robert Haas, David Steele, Chris Browne, Jim Nasby, Josh Berkus, Masao Fujii, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Andrew Dunstan, Jose Luis Tallon, Yurie Enomoto, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Yasuo Honda, Dave Cramer,Steve Singer, Alexander Korotkov, Oleg Bartunov,Konstantin Knizhnik, Chris Malek, Ed Espino, Keith Fiske<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Noah Misch, Tetsuo Sakata, Naoya Anzai, Robert Haas, Jim Nasby, Josh Berkus, Chris Winters, Ingmar Alting, Mehmet Emin KARAKAŞ, Jose Luis Tallon, Oleg Bartunov, Konstanti Knizhnik, Chris Malek, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductors - GPU, SSD, NVRAM, FPGA, PMEM...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|Matthew Wilcox, Josh Berkus, Satoshi Nagayasu, Jose Luis Tallon, Naoya Anzai, Mack McCauley, Shigeru HANADA, Michael Robinson, Ingmar Alting<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Columnar Storage<br />
|Open<br />
|<br />
|Álvaro Herrera<br />
|Ozgun Erdogan, Tomas Vondra, KaiGai Kohei, Amit Kapila, Josh Berkus, Naoya Anzai, Amit Langote, Robert Haas, David Steele, Rod Taylor, Chris Browne, Jim Nasby, Chris Winters, Nat Wyatt, Masao Fujii, Fabrízio de Royes Mello, Euler Taveira, Satoshi Nagayasu, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Heikki Linnakangas, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Andres Freund, Marc Jeanneret, Marco Slot, Ed Espino, Arul Shaji, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis, Amit Langote, Kumar Rajeev Rastogi, Josh Berkus, Simon Riggs, Robert Haas, Jim Nasby, Masao Fujii, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Yurie Enomoto, Masahiko Sawada, Shigeru HANADA, Yasuo Honda,Steve Singer, Marc Jeanneret<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|Kumar Rajeev Rastogi, Noah Misch, Ilya Kosmodemiansky, Amit Kapila, Simon Riggs, Rod Taylor, Jim Nasby, Josh Berkus, Nat Wyatt, Christophe Pettus, Satoshi Nagayasu<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Slony Development]]<br />
|Open<br />
|<br />
| Steve Singer, Chris Browne, Jan Wieck<br />
| Josh Berkus, Rod Taylor, Jim Nasby, Satoshi Nagayasu, Yurie Enomoto<br />
<br />
|- style="background-color:lightgray;"<br />
|[[DockerizingPostgres|Dockerizing Postgres]]<br />
|Open<br />
|<br />
| Josh Berkus<br />
| Simon Riggs, Nat Wyatt, Christophe Pettus, Fabrízio de Royes Mello, Jan Urbański, Michael Robinson, Jeff Davis, Rob Young, Greg Smith, Keith Fiske<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Bi Directional Replication & Logical Decoding|BDR]]<br />
|Open<br />
|<br />
| Simon Riggs<br />
| Andres Freund, Jim Nasby, Josh Berkus, Mehmet Emin KARAKAŞ, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Michael Robinson, Dave Cramer,Steve Singer, Jeff Davis, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|[[AutonomousTransactionsUnconference2015|Autonomous Transactions]]<br />
|Open<br />
|<br />
| Simon Riggs, Kumar Rajeev Rastogi<br />
| David Steele, Jim Nasby, Josh Berkus, Nat Wyatt, Masao Fujii, Euler Taveira, Andrew Dunstan, Masahiko Sawada, Michael Robinson, Amit Kapila, Chris Malek, Arul Shaji, Fabrízio de Royes Mello<br />
<br />
|- style="background-color:lightgray;"<br />
|Audit Logging<br />
|Open<br />
|<br />
| David Steele<br />
| Josh Berkus, Nat Wyatt, Masao Fujii, Christophe Pettus, Fabio Telles, Satoshi Nagayasu, Yurie Enomoto, Mack McCauley, Masahiko Sawada, Michael Robinson, Oleg Bartunov, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pg_shard v2.0 and Lessons Learned from NoSQL Databases ]]<br />
|Open<br />
|<br />
| Ozgun Erdogan, Marco Slot <br />
| Josh Berkus, Jim Nasby, Josh Berkus, Chris Winters, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Satoshi Nagayasu, Shigeru HANADA, Michael Robinson, Oleg Bartunov, Chris Malek<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Direction of json and jsonb<br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Christophe Pettus, Masahiko Sawada, Michael Robinson, Rod Taylor, Alexander Korotkov, Oleg Bartunov, Chris Malek, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Sparse Set Type <br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Michael Robinson<br />
<br />
|- style="background-color:lightgray;"<br />
|Testing Framework Adequacy<br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Christophe Pettus, Mack McCauley, Michael Robinson,Steve Singer, Oleg Bartunov, Ed Espino<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Vertical Scalability w.r.t Writes ==<br />
Purpose of this discussion:<br />
* Discuss about priority/importance of various performance and scalability problems<br />
* Solution/Idea to solve most important problem('s)<br />
* Is pgbench sufficient to capture various kind of real world workloads?<br />
<br />
Some of important performance problems I have in mind are:<br />
* Avoid/Reduce Vacuum Freeze<br />
* Bloat<br />
Heap<br />
Index<br />
* Instability in TPS due to checkpointer flush<br />
* Tuple size<br />
Heap Tuple Header <br />
Alignment in index can lead to bigger index size for simple datatypes<br />
Scalability bottlenecks<br />
* Locks<br />
ProcArrayLock<br />
WALWriteLock<br />
CLOGControlLock<br />
Lock for Relation Extension<br />
<br />
* Writes, especially when data doesn't fit in shared buffers.<br />
Write Performance<br />
Double Buffering<br />
In-memory table/tablespaces<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* This will be, ehem, secure so nothing will be written here<br />
<br />
== Partitioning ==<br />
Proposal to enhance partitioning support in PostgreSQL was posted to -hackers last year and resulted in discussion of some ideas regarding implementation. Late in the discussion, a crude WIP patch was also posted with some experimental syntax, catalog changes, an idea for internal representation and a proof-of-concept INSERT tuple routing function demonstrating practicality of the internal representation. It would be nice to carry the discussion forward at the same time implementing a patch to be proposed, reviewed early in the 9.6 development cycle. Points to discuss could be: <br />
<br />
* New features and old inheritance based implementation<br />
* Planner considerations for new partitioned table<br />
* Need for a new Append-like executor node for partitioned tables<br />
* DML/DDL restrictions on partitioned tables and partitions<br />
* Basically any considerations for partitioned tables and partitions that are explicitly defined so at a layer that's above the storage layer<br />
* Other points that come up<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductors ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* Differences in APIs between PMEM and SSD/Disk - Unlike traditional device, PMEM assumes memory mapped cacheline based interface. PostgreSQL's storage manager is a good candidate for investigation to support both type of devices; one is traditional VFS based one, other is memory mapped one. PMEM library will offer clear interface to applications.<br />
* If all system RAM is persistent, we don't need transaction log, however, we are in the meantime of technology improvement. In the near future, we utilize PMEM on a particular portion, like transaction log buffer.<br />
* Does GPU help data encryption/description? Yes, it may be possible, probably, special code shall implement type input/output handler. However, a certain level of complexity of calculation is needed to make sense GPU acceleration.<br />
* In PG-Strom case, GPU works well to the type of workload to reduce number of rows; scan, join (with equal condition) or aggregation.<br />
* PL function is predefined, so it may work fine even if HDL build and device rewrite takes 30minutes.<br />
* XeonPhi needs another optimization than GPU. Because GPU logic usually has inter-thread-synchronization, however, it is emulated by software on XeonPhi. It is too challenging to write once, run anywhere,<br />
* GPU and AVX (SIMD operation) will work ideally on column oriented data structure, so Alvaro's work is very desirable from the standpoint of HW acceleration.<br />
<br />
[https://wiki.postgresql.org/images/6/66/20150617_PGconf2015_Unconf_KaiGai.pdf discussion material]<br />
<br />
* Action items<br />
** investigation of storage manager API to fit both of VFS and PMEM<br />
** columnar storage as better infrastructure of HW acceleration<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Native Columnar Storage ==<br />
<br />
See Alvaro's [http://www.postgresql.org/message-id/20150611230316.GM133018@postgresql.org email to Hackers].<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Audit Logging ==<br />
<br />
Audit logging is an important part of a RDBMS for many users and applications. Discuss how best to incorporate audit logging into PostgreSQL and what must be included at a minimum to make the feature viable. <br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Direction of json and jsonb ==<br />
<br />
What are the future needs of the JSON types? Recent suggestions have included an indexable "exists" operator, the JSON pointer and JSON patch standards, <br />
recursive merge, intersection, and being able to assign to a subdocument (json#>path as an lvalue). What are people using these types for, and what are <br />
the major gaps in functionality?<br />
<br />
=== Meeting Notes ===<br />
<br />
* Tom: need to work on selectivity estimators for the indexable operators<br />
* Oleg: update syntax wanted<br />
** UPDATE table SET jsoncol['a'][1]['b'] = ...jsonvalue...<br />
** similar syntax for fetching a value<br />
** potentially, most container types should be able to offer array-like fetch and set syntax<br />
* Oleg: jsquery-like query language<br />
** there will be a talk about that tomorrow<br />
** possible patch in 9.7 timeframe<br />
* Oleg: validation of JSON values against a schema<br />
** is there any accepted standard for JSON schemas?<br />
<br />
=== Attendees ===<br />
* Tom Lane<br />
<br />
== Native Sparse Set Type ==<br />
<br />
Sets over small domains can be reasonably modeled by bitmaps, but sets over very large domains can not.<br />
Is there a need for such sets? How would we implement them? Arrays? Balanced trees? Something else?<br />
What types of sets would we allow? Anything with Btree operators, or more restricted? What would the notation look like?<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Testing Framework Adequacy ==<br />
<br />
The buildfarm is more than 10 years old, and the testing needs of Postgres and its ofware ecosystem have changed radically in that time.<br />
What do we now need in the way of testing? How do we test complex arrangements such as the various sorts of replication in an automated way?<br />
Do we need a new framwork, or can the existing framework be adapted to our needs?<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
<br />
== Unconference schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!DMS1110<br />
!DMS1120<br />
!DMS1160<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 13:30-14:30<br />
|<br />
|PostgreSQL and SMR Drives - the future of magnetic storage means very expensive random writes<br />
|pgPool2 towards version 3.5<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 14:45-15:45<br />
|Autonomous Transactions<br />
|Native Compilation + LLVM<br />
|pgAdmin4<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 16:00-17:30<br />
|Bi Directional Replication & Logical Decoding&#124;BDR<br />
|<br />
|Advocacy Team Meeting<br />
<br />
|- style="background-color:lightgray;"<br />
|<br />
|<br />
|<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 10:00-11:15<br />
|Security Team Meeting<br />
|Slony Development<br />
|Utilization of modern semiconductors - GPU, SSD, NVRAM, FPGA, PMEM...<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 11:30-13:00<br />
|Direction of json and jsonb<br />
|Dockerizing Postgres<br />
|Vertical Scalability w.r.t Write<br />
<br />
|- style="background-color:lightgray;"<br />
|Lunch time<br />
|PGCAC Board Meeting 2015<br />
|<br />
|LUNCH!!<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 14:00-15:00<br />
|Native Sparse Set Type<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Horizontal Scalability / Sharding in PostgreSQL - ground covered so far and remaining to be covered.<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 15:15-16:15<br />
|pg_shard v2.0 and Lessons Learned from NoSQL Databases<br />
|Testing Framework Adequacy<br />
|Native Columnar Storage<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 16:30-17:30<br />
|Foreign Data Wrapper enhancements<br />
|Audit Logging<br />
|Partitioning<br />
<br />
|- style="background-color:lightgray;"<br />
|17:30<br />
|<br />
|<br />
|Picture!</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=25191PgCon 2015 Developer Unconference2015-06-17T15:56:28Z<p>Kaigai: /* Utilization of modern semiconductors */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
== Schedule ==<br />
<br />
[[PgCon_2015_Developer_Unconference_Schedule|Schedule is here]]. It is updated frequently.<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
# Ingmar Alting<br />
# Naoya Anzai (arrive tuesday evening)<br />
# Chris Autry<br />
# Ashutosh Bapat<br />
# Oleg Bartunov<br />
# Josh Berkus<br />
# Christopher Browne<br />
# Joe Conway<br />
# Jeff Davis<br />
# Andrew Dunstan<br />
# Yurie Enomoto<br />
# Ozgun Erdogan (Wednesday)<br />
# Ed Espino<br />
# Andres Freund<br />
# Stephen Frost<br />
# Masao Fujii<br />
# Etsuro Fujita<br />
# Peter Geoghegan<br />
# Kevin Grittner<br />
# Robert Haas<br />
# Ahsan Hadi<br />
# Magnus Hagander<br />
# Shigeru Hanada<br />
# Álvaro Herrera<br />
# Yasuo Honda<br />
# Kyotaro Horiguchi<br />
# Thierry Husson (Wednesday @ 11am)<br />
# Ayumi Ishii<br />
# Tatsuo Ishii<br />
# Moshe Jacobson<br />
# Stefan Kaltenbrunner<br />
# Amit Kapila<br />
# Mehmet Emin KARAKAŞ<br />
# Motoyuki Kawaba (arrive Tuesday evening)<br />
# Konstantin Knizhnik<br />
# KaiGai Kohei (arrive tuesday evening)<br />
# Alexander Korotkov<br />
# Ilya Kosmodemiansky<br />
# Dilip Kumar<br />
# Tom Lane<br />
# Amit Langote<br />
# Heikki Linnakangas<br />
# Chris Malek (Wednesday @ 11am)<br />
# Grant McAlister<br />
# Mack McCauley<br />
# Fabrízio de Royes Mello<br />
# Noah Misch<br />
# Bruce Momjian<br />
# Yugo Nagata<br />
# Satoshi Nagayasu<br />
# Jim Nasby<br />
# Dave Page<br />
# Christophe Pettus<br />
# Tyler Poland<br />
# Paul Ramsey<br />
# Kumar Rajeev Rastogi<br />
# Simon Riggs<br />
# Michael Robinson<br />
# Tetsuo Sakata<br />
# Masahiko Sawada<br />
# Arul Shaji<br />
# Dan Shuster<br />
# Steve Singer (arrive tuesday mid-afternoon)<br />
# Marco Slot <br />
# Greg Smith<br />
# David Steele (arrive tuesday evening)<br />
# Jose Luis Tallon (arrives tuesday evening)<br />
# Yasin TATAR<br />
# Euler Taveira<br />
# Rod Taylor<br />
# Fabio Telles<br />
# Jan Urbański (Wednesday)<br />
# Tomas Vondra<br />
# Jan Wieck (arrive tuesday evening)<br />
# Chris Winters<br />
# Nat Wyatt<br />
# Jimmy Yih<br />
# Rob Young<br />
<br />
=Topics=<br />
<br />
''See the schedule at [[PgCon 2015 Developer Unconference Schedule]]''<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|Picture!<br />
|Open<br />
|Oleg Bartunov (Fuji 16-55)<br />
|All!<br />
|All!<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander, Joe Conway, David Steele, Fabrízio de Royes Mello, Satoshi Nagayasu, Dave Cramer, Alexander Korotkov, Chris Malek, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus, Joe Conway, Michael Robinson, Oleg Bartunov, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|Amit Kapila<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen, Ilya Kosmodemiansky, Tomas Vondra, Grant McAlister, Joe Conway, Peter Geoghegan, Kyotaro Horiguchi, Simon Riggs, Amit Langote, Andres Freund, Robert Haas, David Steele, Rod Taylor, Jim Nasby, Chris Winters, Nat Wyatt, Noah Misch, Masao Fujii, Mehmet Emin KARAKAŞ, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Andrew Dunstan, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Dave Cramer, Steve Singer, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Marc Jeanneret, Chris Malek<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|Noah Misch, Álvaro Herrera, Andres Freund, Robert Haas, Tom Lane, Andrew Dunstan,Oleg Bartunov<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis, Ozgun Erdogan, Tomas Vondra, Peter Geoghegan, Robert Haas, Chris Browne, Josh Berkus, Ingmar Alting, Masao Fujii, Christophe Pettus, Jose Luis Tallon, Heikki Linnakangas, Alexander Korotkov,Oleg Bartunov,Konstantin Knizhnik<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Tetsuo Sakata, Simon Riggs, Robert Haas, David Steele, Rod Taylor, Chris Browne, Jim Nasby, Josh Berkus, Chris Winters, Masao Fujii, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Satoshi Nagayasu, Andrew Dunstan, Mack McCauley, Shigeru HANADA, Michael Robinson, Steve Singer, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Andres Freund, Marc Jeanneret, Chris Malek, Marco Slot<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Closed<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|During Lunch Wed.<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat, Ahsan Hadi, Yurie Enomoto, Chris Malek<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Kyotaro Horiguchi, KaiGai Kohei, Noah Misch, Tetsuo Sakata, Peter Geoghegan, Álvaro Herrera, Thierry Husson, Joe Conway, Naoya Anzai, Robert Haas, David Steele, Chris Browne, Jim Nasby, Josh Berkus, Masao Fujii, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Andrew Dunstan, Jose Luis Tallon, Yurie Enomoto, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Yasuo Honda, Dave Cramer,Steve Singer, Alexander Korotkov, Oleg Bartunov,Konstantin Knizhnik, Chris Malek, Ed Espino, Keith Fiske<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Noah Misch, Tetsuo Sakata, Naoya Anzai, Robert Haas, Jim Nasby, Josh Berkus, Chris Winters, Ingmar Alting, Mehmet Emin KARAKAŞ, Jose Luis Tallon, Oleg Bartunov, Konstanti Knizhnik, Chris Malek, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductors - GPU, SSD, NVRAM, FPGA, PMEM...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|Matthew Wilcox, Josh Berkus, Satoshi Nagayasu, Jose Luis Tallon, Naoya Anzai, Mack McCauley, Shigeru HANADA, Michael Robinson, Ingmar Alting<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Columnar Storage<br />
|Open<br />
|<br />
|Álvaro Herrera<br />
|Ozgun Erdogan, Tomas Vondra, KaiGai Kohei, Amit Kapila, Josh Berkus, Naoya Anzai, Amit Langote, Robert Haas, David Steele, Rod Taylor, Chris Browne, Jim Nasby, Chris Winters, Nat Wyatt, Masao Fujii, Fabrízio de Royes Mello, Euler Taveira, Satoshi Nagayasu, Mack McCauley, Masahiko Sawada, Shigeru HANADA, Michael Robinson, Heikki Linnakangas, Alexander Korotkov, Oleg Bartunov, Konstantin Knizhnik, Andres Freund, Marc Jeanneret, Marco Slot, Ed Espino, Arul Shaji, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis, Amit Langote, Kumar Rajeev Rastogi, Josh Berkus, Simon Riggs, Robert Haas, Jim Nasby, Masao Fujii, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Fabio Telles, Yurie Enomoto, Masahiko Sawada, Shigeru HANADA, Yasuo Honda,Steve Singer, Marc Jeanneret<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|Kumar Rajeev Rastogi, Noah Misch, Ilya Kosmodemiansky, Amit Kapila, Simon Riggs, Rod Taylor, Jim Nasby, Josh Berkus, Nat Wyatt, Christophe Pettus, Satoshi Nagayasu<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Slony Development]]<br />
|Open<br />
|<br />
| Steve Singer, Chris Browne, Jan Wieck<br />
| Josh Berkus, Rod Taylor, Jim Nasby, Satoshi Nagayasu, Yurie Enomoto<br />
<br />
|- style="background-color:lightgray;"<br />
|[[DockerizingPostgres|Dockerizing Postgres]]<br />
|Open<br />
|<br />
| Josh Berkus<br />
| Simon Riggs, Nat Wyatt, Christophe Pettus, Fabrízio de Royes Mello, Jan Urbański, Michael Robinson, Jeff Davis, Rob Young, Greg Smith, Keith Fiske<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Bi Directional Replication & Logical Decoding|BDR]]<br />
|Open<br />
|<br />
| Simon Riggs<br />
| Andres Freund, Jim Nasby, Josh Berkus, Mehmet Emin KARAKAŞ, Christophe Pettus, Fabrízio de Royes Mello, Euler Taveira, Michael Robinson, Dave Cramer,Steve Singer, Jeff Davis, Arul Shaji<br />
<br />
|- style="background-color:lightgray;"<br />
|[[AutonomousTransactionsUnconference2015|Autonomous Transactions]]<br />
|Open<br />
|<br />
| Simon Riggs, Kumar Rajeev Rastogi<br />
| David Steele, Jim Nasby, Josh Berkus, Nat Wyatt, Masao Fujii, Euler Taveira, Andrew Dunstan, Masahiko Sawada, Michael Robinson, Amit Kapila, Chris Malek, Arul Shaji, Fabrízio de Royes Mello<br />
<br />
|- style="background-color:lightgray;"<br />
|Audit Logging<br />
|Open<br />
|<br />
| David Steele<br />
| Josh Berkus, Nat Wyatt, Masao Fujii, Christophe Pettus, Fabio Telles, Satoshi Nagayasu, Yurie Enomoto, Mack McCauley, Masahiko Sawada, Michael Robinson, Oleg Bartunov, Tyler Poland<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pg_shard v2.0 and Lessons Learned from NoSQL Databases ]]<br />
|Open<br />
|<br />
| Ozgun Erdogan, Marco Slot <br />
| Josh Berkus, Jim Nasby, Josh Berkus, Chris Winters, Mehmet Emin KARAKAŞ, Fabrízio de Royes Mello, Satoshi Nagayasu, Shigeru HANADA, Michael Robinson, Oleg Bartunov, Chris Malek<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Direction of json and jsonb<br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Christophe Pettus, Masahiko Sawada, Michael Robinson, Rod Taylor, Alexander Korotkov, Oleg Bartunov, Chris Malek, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Sparse Set Type <br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Michael Robinson<br />
<br />
|- style="background-color:lightgray;"<br />
|Testing Framework Adequacy<br />
|Open<br />
|<br />
| Andrew Dunstan<br />
| Josh Berkus, Christophe Pettus, Mack McCauley, Michael Robinson,Steve Singer, Oleg Bartunov, Ed Espino<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Vertical Scalability w.r.t Writes ==<br />
Purpose of this discussion:<br />
* Discuss about priority/importance of various performance and scalability problems<br />
* Solution/Idea to solve most important problem('s)<br />
* Is pgbench sufficient to capture various kind of real world workloads?<br />
<br />
Some of important performance problems I have in mind are:<br />
* Avoid/Reduce Vacuum Freeze<br />
* Bloat<br />
Heap<br />
Index<br />
* Instability in TPS due to checkpointer flush<br />
* Tuple size<br />
Heap Tuple Header <br />
Alignment in index can lead to bigger index size for simple datatypes<br />
Scalability bottlenecks<br />
* Locks<br />
ProcArrayLock<br />
WALWriteLock<br />
CLOGControlLock<br />
Lock for Relation Extension<br />
<br />
* Writes, especially when data doesn't fit in shared buffers.<br />
Write Performance<br />
Double Buffering<br />
In-memory table/tablespaces<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* This will be, ehem, secure so nothing will be written here<br />
<br />
== Partitioning ==<br />
Proposal to enhance partitioning support in PostgreSQL was posted to -hackers last year and resulted in discussion of some ideas regarding implementation. Late in the discussion, a crude WIP patch was also posted with some experimental syntax, catalog changes, an idea for internal representation and a proof-of-concept INSERT tuple routing function demonstrating practicality of the internal representation. It would be nice to carry the discussion forward at the same time implementing a patch to be proposed, reviewed early in the 9.6 development cycle. Points to discuss could be: <br />
<br />
* New features and old inheritance based implementation<br />
* Planner considerations for new partitioned table<br />
* Need for a new Append-like executor node for partitioned tables<br />
* DML/DDL restrictions on partitioned tables and partitions<br />
* Basically any considerations for partitioned tables and partitions that are explicitly defined so at a layer that's above the storage layer<br />
* Other points that come up<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductors ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* Differences in APIs between PMEM and SSD/Disk - Unlike traditional device, PMEM assumes memory mapped cacheline based interface. PostgreSQL's storage manager is a good candidate for investigation to support both type of devices; one is traditional VFS based one, other is memory mapped one. PMEM library will offer clear interface to applications.<br />
* If all system RAM is persistent, we don't need transaction log, however, we are in the meantime of technology improvement. In the near future, we utilize PMEM on a particular portion, like transaction log buffer.<br />
* Does GPU help data encryption/description? Yes, it may be possible, probably, special code shall implement type input/output handler. However, a certain level of complexity of calculation is needed to make sense GPU acceleration.<br />
* In PG-Strom case, GPU works well to the type of workload to reduce number of rows; scan, join (with equal condition) or aggregation.<br />
* PL function is predefined, so it may work fine even if HDL build and device rewrite takes 30minutes.<br />
* XeonPhi needs another optimization than GPU. Because GPU logic usually has inter-thread-synchronization, however, it is emulated by software on XeonPhi. It is too challenging to write once, run anywhere,<br />
* GPU and AVX (SIMD operation) will work ideally on column oriented data structure, so Alvaro's work is very desirable from the standpoint of HW acceleration.<br />
<br />
[https://wiki.postgresql.org/images/6/66/20150617_PGconf2015_Unconf_KaiGai.pdf discussion material]<br />
<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Native Columnar Storage ==<br />
<br />
See Alvaro's [http://www.postgresql.org/message-id/20150611230316.GM133018@postgresql.org email to Hackers].<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Audit Logging ==<br />
<br />
Audit logging is an important part of a RDBMS for many users and applications. Discuss how best to incorporate audit logging into PostgreSQL and what must be included at a minimum to make the feature viable. <br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Direction of json and jsonb ==<br />
<br />
What are the future needs of the JSON types? Recent suggestions have included an indexable "exists" operator, the JSON pointer and JSON patch standards, <br />
recursive merge, intersection, and being able to assign to a subdocument (json#>path as an lvalue). What are people using these types for, and what are <br />
the major gaps in functionality?<br />
<br />
=== Meeting Notes ===<br />
<br />
* Tom: need to work on selectivity estimators for the indexable operators<br />
* Oleg: update syntax wanted<br />
** UPDATE table SET jsoncol['a'][1]['b'] = ...jsonvalue...<br />
** similar syntax for fetching a value<br />
** potentially, most container types should be able to offer array-like fetch and set syntax<br />
* Oleg: jsquery-like query language<br />
** there will be a talk about that tomorrow<br />
** possible patch in 9.7 timeframe<br />
* Oleg: validation of JSON values against a schema<br />
** is there any accepted standard for JSON schemas?<br />
<br />
=== Attendees ===<br />
* Tom Lane<br />
<br />
== Native Sparse Set Type ==<br />
<br />
Sets over small domains can be reasonably modeled by bitmaps, but sets over very large domains can not.<br />
Is there a need for such sets? How would we implement them? Arrays? Balanced trees? Something else?<br />
What types of sets would we allow? Anything with Btree operators, or more restricted? What would the notation look like?<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Testing Framework Adequacy ==<br />
<br />
The buildfarm is more than 10 years old, and the testing needs of Postgres and its ofware ecosystem have changed radically in that time.<br />
What do we now need in the way of testing? How do we test complex arrangements such as the various sorts of replication in an automated way?<br />
Do we need a new framwork, or can the existing framework be adapted to our needs?<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
<br />
== Unconference schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!DMS1110<br />
!DMS1120<br />
!DMS1160<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 13:30-14:30<br />
|<br />
|PostgreSQL and SMR Drives - the future of magnetic storage means very expensive random writes<br />
|pgPool2 towards version 3.5<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 14:45-15:45<br />
|Autonomous Transactions<br />
|Native Compilation + LLVM<br />
|pgAdmin4<br />
<br />
|- style="background-color:lightgray;"<br />
|Tue 16:00-17:30<br />
|Bi Directional Replication & Logical Decoding&#124;BDR<br />
|<br />
|Advocacy Team Meeting<br />
<br />
|- style="background-color:lightgray;"<br />
|<br />
|<br />
|<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 10:00-11:15<br />
|Security Team Meeting<br />
|Slony Development<br />
|Utilization of modern semiconductors - GPU, SSD, NVRAM, FPGA, PMEM...<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 11:30-13:00<br />
|Direction of json and jsonb<br />
|Dockerizing Postgres<br />
|Vertical Scalability w.r.t Write<br />
<br />
|- style="background-color:lightgray;"<br />
|Lunch time<br />
|PGCAC Board Meeting 2015<br />
|<br />
|LUNCH!!<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 14:00-15:00<br />
|Native Sparse Set Type<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Horizontal Scalability / Sharding in PostgreSQL - ground covered so far and remaining to be covered.<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 15:15-16:15<br />
|pg_shard v2.0 and Lessons Learned from NoSQL Databases<br />
|Testing Framework Adequacy<br />
|Native Columnar Storage<br />
<br />
|- style="background-color:lightgray;"<br />
|Wed 16:30-17:30<br />
|Foreign Data Wrapper enhancements<br />
|Audit Logging<br />
|Partitioning<br />
<br />
|- style="background-color:lightgray;"<br />
|17:30<br />
|<br />
|<br />
|Picture!</div>Kaigaihttps://wiki.postgresql.org/index.php?title=File:20150617_PGconf2015_Unconf_KaiGai.pdf&diff=25179File:20150617 PGconf2015 Unconf KaiGai.pdf2015-06-17T15:37:14Z<p>Kaigai: Discussion material KaiGai used in the PGcon2015 Unconference</p>
<hr />
<div>Discussion material KaiGai used in the PGcon2015 Unconference</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24928PgCon 2015 Developer Unconference2015-06-11T03:52:42Z<p>Kaigai: /* RSVPs */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Ashutosh Bapat<br />
* Oleg Bartunov<br />
* Josh Berkus<br />
* Jeff Davis<br />
* Ozgun Erdogan<br />
* Stephen Frost<br />
* Etsuro Fujita<br />
* Kevin Grittner<br />
* Robert Haas<br />
* Ahsan Hadi<br />
* Shigeru Hanada<br />
* Álvaro Herrera<br />
* Kyotaro Horiguchi<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Konstantin Knizhnik<br />
* KaiGai Kohei (arrive tuesday evening)<br />
* Alexander Korotkov<br />
* Ilya Kosmodemiansky<br />
* Amit Langote<br />
* Grant McAlister<br />
* Noah Misch<br />
* Bruce Momjian<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Tetsuo Sakata<br />
* Marco Slot (Wednesday)<br />
* Greg Smith<br />
* Steve Singer (arrive tuesday mid-afternoon)<br />
* Tomas Vondra<br />
* Tatsuo Ishii<br />
* Yugo Nagata<br />
* Jan Wieck (arrive tuesday evening)<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen, Ilya Kosmodemiansky, Tomas Vondra<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|Noah Misch<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis, Ozgun Erdogan, Tomas Vondra<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Tetsuo Sakata<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Kyotaro Horiguchi, KaiGai Kohei, Noah Misch, Tetsuo Sakata<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Noah Misch, Tetsuo Sakata<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, FPGA, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Columnar Storage<br />
|Open<br />
|<br />
|Álvaro Herrera<br />
|Ozgun Erdogan, Tomas Vondra, KaiGai Kohei<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis, Amit Langote, Kumar Rajeev Rastogi<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|Kumar Rajeev Rastogi, Noah Misch, Ilya Kosmodemiansky<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Slony Development]]<br />
|Open<br />
|<br />
| Steve Singer, Chris Browne, Jan Wieck<br />
| <br />
<br />
|- style="background-color:lightgray;"<br />
|[[DockerizingPostgres|Dockerizing Postgres]]<br />
|<br />
|<br />
| Josh Berkus<br />
| <br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
Proposal to enhance partitioning support in PostgreSQL was posted to -hackers last year and resulted in discussion of some ideas regarding implementation. Late in the discussion, a crude WIP patch was also posted with some experimental syntax, catalog changes, an idea for internal representation and a proof-of-concept INSERT tuple routing function demonstrating practicality of the internal representation. It would be nice to carry the discussion forward at the same time implementing a patch to be proposed, reviewed early in the 9.6 development cycle. Points to discuss could be: <br />
<br />
* New features and old inheritance based implementation<br />
* Planner considerations for new partitioned table<br />
* Need for a new Append-like executor node for partitioned tables<br />
* DML/DDL restrictions on partitioned tables and partitions<br />
* Basically any considerations for partitioned tables and partitions that are explicitly defined so at a layer that's above the storage layer<br />
* Other points that come up<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductor ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24916PgCon 2015 Developer Unconference2015-06-10T14:36:25Z<p>Kaigai: </p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Ashutosh Bapat<br />
* Oleg Bartunov<br />
* Josh Berkus<br />
* Jeff Davis<br />
* Ozgun Erdogan<br />
* Stephen Frost<br />
* Kevin Grittner<br />
* Robert Haas<br />
* Álvaro Herrera<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Konstantin Knizhnik<br />
* Alexander Korotkov<br />
* Ilya Kosmodemiansky<br />
* Amit Langote<br />
* Grant McAlister<br />
* Noah Misch<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Marco Slot<br />
* Greg Smith<br />
* KaiGai Kohei<br />
* Tomas Vondra<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen, Ilya Kosmodemiansky, Tomas Vondra<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|Noah Misch<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis, Ozgun Erdogan, Tomas Vondra<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen, Jeff Davis, Amit Langote, Kyotaro Horiguchi<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Kyotaro Horiguchi, KaiGai Kohei, Noah Misch<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Amit Langote, Kyotaro Horiguchi, Noah Misch<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, FPGA, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Columnar Storage<br />
|Open<br />
|<br />
|Álvaro Herrera<br />
|Ozgun Erdogan, Tomas Vondra, KaiGai Kohei<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis, Amit Langote, Kumar Rajeev Rastogi<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|Kumar Rajeev Rastogi, Noah Misch, Ilya Kosmodemiansky<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Slony Development]]<br />
|Open<br />
|<br />
| Steve Singer, Chris Browne, Jan Wieck<br />
| <br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
Proposal to enhance partitioning support in PostgreSQL was posted to -hackers last year and resulted in discussion of some ideas regarding implementation. Late in the discussion, a crude WIP patch was also posted with some experimental syntax, catalog changes, an idea for internal representation and a proof-of-concept INSERT tuple routing function demonstrating practicality of the internal representation. It would be nice to carry the discussion forward at the same time implementing a patch to be proposed, reviewed early in the 9.6 development cycle. Points to discuss could be: <br />
<br />
* New features and old inheritance based implementation<br />
* Planner considerations for new partitioned table<br />
* Need for a new Append-like executor node for partitioned tables<br />
* DML/DDL restrictions on partitioned tables and partitions<br />
* Basically any considerations for partitioned tables and partitions that are explicitly defined so at a layer that's above the storage layer<br />
* Other points that come up<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductor ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24846PgCon 2015 Developer Unconference2015-06-06T05:31:37Z<p>Kaigai: </p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Ashutosh Bapat<br />
* Oleg Bartunov<br />
* Josh Berkus<br />
* Jeff Davis<br />
* Stephen Frost<br />
* Robert Haas<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Konstantin Knizhnik<br />
* Alexander Korotkov<br />
* Amit Langote<br />
* Grant McAlister<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Greg Smith<br />
* KaiGai Kohei<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen, Jeff Davis, Amit Langote, Kyotaro Horiguchi<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Kyotaro Horiguchi, KaiGai Kohei<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Amit Langote, Kyotaro Horiguchi<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, FPGA, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis, Amit Langote<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
Proposal to enhance partitioning support in PostgreSQL was posted to -hackers last year and resulted in discussion of some ideas regarding implementation. Late in the discussion, a crude WIP patch was also posted with some experimental syntax, catalog changes, an idea for internal representation and a proof-of-concept INSERT tuple routing function demonstrating practicality of the internal representation. It would be nice to carry the discussion forward at the same time implementing a patch to be proposed, reviewed early in the 9.6 development cycle. Points to discuss could be: <br />
<br />
* New features and old inheritance based implementation<br />
* Planner considerations for new partitioned table<br />
* Need for a new Append-like execution node for partitioned tables<br />
* DML/DDL restrictions on new partitioned tables and partitions<br />
* Other points that come up<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductor ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24836PgCon 2015 Developer Unconference2015-06-05T05:00:12Z<p>Kaigai: /* RSVPs */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Ashutosh Bapat<br />
* Oleg Bartunov<br />
* Josh Berkus<br />
* Jeff Davis<br />
* Stephen Frost<br />
* Robert Haas<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Konstantin Knizhnik<br />
* Alexander Korotkov<br />
* Amit Langote<br />
* Grant McAlister<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Greg Smith<br />
* KaiGai Kohei<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen, Jeff Davis, Amit Langote<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis, Amit Langote<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, FPGA, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis, Amit Langote<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductor ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24832PgCon 2015 Developer Unconference2015-06-05T03:42:51Z<p>Kaigai: /* Utilization of modern semiconductor */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Ashutosh Bapat<br />
* Oleg Bartunov<br />
* Josh Berkus<br />
* Jeff Davis<br />
* Stephen Frost<br />
* Robert Haas<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Konstantin Knizhnik<br />
* Alexander Korotkov<br />
* Amit Langote<br />
* Grant McAlister<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Greg Smith<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|Table partitioning - WIP, direction<br />
|Open<br />
|<br />
|Amit Langote<br />
|Ashutosh Bapat, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, FPGA, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductor ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get the future direction in short and middle/long term.<br />
<br />
* GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
<br />
* SSD, NVRAM - likely, game changer of storage layer on both of read/write workloads. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24831PgCon 2015 Developer Unconference2015-06-05T03:40:44Z<p>Kaigai: </p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Ashutosh Bapat<br />
* Oleg Bartunov<br />
* Josh Berkus<br />
* Jeff Davis<br />
* Stephen Frost<br />
* Robert Haas<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Konstantin Knizhnik<br />
* Alexander Korotkov<br />
* Amit Langote<br />
* Grant McAlister<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Greg Smith<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith, Hannu Valtonen<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat, Etsuro Fujita<br />
|Hannu Valtonen<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|Table partitioning - WIP, direction<br />
|Open<br />
|<br />
|Amit Langote<br />
|Ashutosh Bapat, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|Ashutosh Bapat<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|Hannu Valtonen, Ashutosh Bapat, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei, Hannu Valtonen, Ashutosh Bapat, Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, FPGA, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Future of PostgreSQL shared-nothing cluster<br />
|Open<br />
|<br />
|Konstantin Knizhnik, Alexander Korotkov, Oleg Bartunov<br />
|Jeff Davis<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PostgreSQL and SMR Drives]] - the future of magnetic storage means very expensive random writes<br />
|Open<br />
|<br />
|Jeff Davis<br />
|<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Utilization of modern semiconductor ==<br />
Recent evolution of semiconductor devices make us re-consider the assumption we stand on, and utilization of its power is key of innovation.<br />
We'd like to have a discussion to get <br />
GPU, FPGA - have advantage on simple but massive amount of calculation. It allows DBMS to perform as data processing platform that works nearby data.<br />
SSD, NVRAM - likely, game changer of storage layer. DBMS also has to pay attention characteristics of these devices.<br />
<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
<br />
== Future of PostgreSQL shared-nothing cluster ==<br />
<br />
=== Meeting Notes ===<br />
In 2015 PostgreSQL Professional company started project of migration PostgreSQL-XL to codebase of PostgreSQL 9.4 and increasing its stability and usability. At this unconference session we'd like to discuss current progress and further development. Generally we'd like to find ways to reduce difference between PostgreSQL and its shared-nothing cluster fork so that burden of the maintenance become manageable. <br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== PostgreSQL and SMR Drives ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24813PgCon 2015 Developer Unconference2015-06-04T02:45:12Z<p>Kaigai: /* Topics */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Josh Berkus<br />
* Stephen Frost<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Amit Langote<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Greg Smith<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat<br />
|Hemmi San<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Table partitioning - WIP, direction<br />
|Open<br />
|<br />
|Amit Langote<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, FPGA, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PgCon_2015_Developer_Unconference&diff=24812PgCon 2015 Developer Unconference2015-06-04T02:34:06Z<p>Kaigai: /* Topics */</p>
<hr />
<div>An Unconference-style multi-track (three tracks are currently planned) event for active PostgreSQL developers will be held from the afternoon of Tuesday 16 June, 2015 through Wednesday 17 June 2015 at the University of Ottawa, as part of PGCon 2015. This Unconference will be focused on technical PostgreSQL development discussions ranging from Clustering and replication to the infrastructure which runs postgresql.org.<br />
<br />
'''Please add your name under RSVPs if you plan to attend.'''<br />
<br />
== Topics ==<br />
<br />
Developers are asked to propose topics which they wish to either present on or which they would like another individual to present on. All topics should be clearly related to PostgreSQL development. The topic should be added to the table below and any required attendees (presumably at least the presenter, and the requester if different) listed. Other attendees of the Unconference who are interested should list themselves as Optional. Note that non-technical topics related to PostgreSQL development will be addressed during the invite-only Developer meeting, being held in advance of the Unconference. Further, the Developer Unconference is for developers of PostgreSQL and user-oriented topics are not appropriate for this venue.<br />
<br />
== Slot assignment ==<br />
<br />
Slots will be assigned based on the topic's interest among the attendees of the Unconference (the number of individuals who listed themselves as attendees). Final determination on any particular topic will be made by the Unconference organizers. Please only participate if you are confident of your attendance at the Unconference.<br />
<br />
== Venue ==<br />
<br />
These meetings will be held at the University of Ottawa. The topics selected, the schedule and the specific room assignments will be published closer to the event and will be based on the information provided here. Please direct any questions to Dave Page (dpage@pgadmin.org).<br />
<br />
== Sponsorship ==<br />
<br />
The Developer Unconference will be sponsored by Salesforce.com, and by NTT Open Source for the Clustering Track.<br />
<br />
== Attendees ==<br />
<br />
While the Unconference is open to all attendees of PGCon, formal invitations will be sent to specific PostgreSQL developers, including the Core team, Major Contributors, Committers, and other developers who have been involved in the 9.4 release. These invitations are intended to encourage developers to attend the Unconference but we are unable to guarantee every invitee a speaking slot.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname):<br />
<br />
* Josh Berkus<br />
* Stephen Frost<br />
* Stefan Kaltenbrunner<br />
* Amit Kapila<br />
* Amit Langote<br />
* Jim Nasby<br />
* Dave Page<br />
* Magnus Hagander<br />
* Kumar Rajeev Rastogi<br />
* Greg Smith<br />
<br />
=Topics=<br />
<br />
'''Please add any topics you wish covered to the table.'''<br />
<br />
'''For any topics you are requesting or presenting on, please add your name in the Required column.'''<br />
<br />
'''For any topics you would like to attend, please add your name in the Interested column.'''<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Topic<br />
!Policy<br />
!Taker of Notes<br />
!Required Attendees<br />
!Interested Attendees<br />
<br />
|- style="background-color:lightgray;"<br />
|pgAdmin4<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost<br />
|Magnus Hagander<br />
<br />
|- style="background-color:lightgray;"<br />
|Infrastructure Q&A<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|WWW Team Meeting<br />
|Open<br />
|<br />
|Dave Page, Stephen Frost, Stefan Kaltenbrunner, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Advocacy Team Meeting<br />
|Open<br />
|<br />
|Stephen Frost<br />
|Magnus Hagander, Greg Smith, Jim Nasby, Josh Berkus<br />
<br />
|- style="background-color:lightgray;"<br />
|Vertical Scalability w.r.t Writes<br />
|Open<br />
|<br />
|Amit Kapila<br />
|Greg Smith<br />
<br />
|- style="background-color:lightgray;"<br />
|Security Team Meeting<br />
|Closed<br />
|<br />
|Heikki Linnakangas, Stephen Frost, Magnus Hagander<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Native Compilation + LLVM<br />
|Open<br />
|<br />
|Kumar Rajeev Rastogi<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Horizontal Scalability / Sharding in PostgreSQL]] - ground covered so far and remaining to be covered. <br />
|Open<br />
|<br />
|Ahsan Hadi, Ashutosh Bapat<br />
|Hemmi San<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Table partitioning - WIP, direction<br />
|Open<br />
|<br />
|Amit Langote<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PGCAC Board Meeting 2015]]<br />
|Open<br />
|Josh Berkus<br />
|Josh Berkus, Chris Browne, Steve Singer, Dan Langille, Dave Page<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|pgPool2 towards version 3.5]]<br />
|Open<br />
|<br />
|Tatsuo Ishii<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Partitioning<br />
|Open<br />
|<br />
|Amit Langote<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|[[PgCon2015ClusterSummit|Foreign Data Wrapper enhancements]]<br />
|Open<br />
|<br />
|Shigeru Hanada, Etsuro Fujita<br />
|KaiGai Kohei<br />
<br />
|- style="background-color:lightgray;"<br />
|Utilization of modern semiconductor - GPU, SSD, NVRAM, ...<br />
|Open<br />
|<br />
|KaiGai Kohei<br />
|<br />
<br />
<br />
|}<br />
<br />
== pgAdmin4 ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Infrastructure Q&A ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== WWW Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Advocacy Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Security Team Meeting ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in<br />
<br />
== Partitioning ==<br />
<br />
=== Meeting Notes ===<br />
* To be filled in<br />
<br />
=== Attendees ===<br />
* To be filled in</div>Kaigaihttps://wiki.postgresql.org/index.php?title=PostgreSQL_9.5_Open_Items&diff=24695PostgreSQL 9.5 Open Items2015-05-18T22:46:19Z<p>Kaigai: /* Open Issues */</p>
<hr />
<div>== Open Issues ==<br />
<br />
* [http://www.postgresql.org/message-id/CAMkU=1z4kq-8+GPounmqqtyRJ-4+Uxf0-2NAzY=Mtq73+g6FrQ@mail.gmail.com docs about max_wal_size in recovery do not match behavior]<br />
* [http://www.postgresql.org/message-id/87d24y7xwa.fsf@news-spur.riddles.org.uk Re: collations in shared catalogs?]<br />
* [http://www.postgresql.org/message-id/55269915.1000309@iki.fi FPW compression leaks information] Make wal_compression SUSET and document potential security risks?<br />
* [http://www.postgresql.org/message-id/20150508223227.GP2523@alvh.no-ip.org "unaddressable bytes" in BRIN]<br />
* [http://www.postgresql.org/message-id/9A28C8860F777E439AA12E8AEA7694F8010DC708@BPXM15GP.gisp.nec.co.jp custom-join has no way to construct Plan nodes of child Path nodes]<br />
<br />
== Resolved Issues ==<br />
<br />
* [http://www.postgresql.org/message-id/546A16EF.9070005@vmware.com BRIN page type identifier] BRIN special space needs reshuffling<br />
* [http://www.postgresql.org/message-id/CAEZATCXHb+tv8YYo4=XRoBzCOywTrM4cncqR57D4ZM7WdFomiQ@mail.gmail.com proposal: searching in array function - array_position] array_offset(s) do not consider arrays not starting from 1<br />
* [http://www.postgresql.org/message-id/CAB7nPqQSdx7coHk0D6G=mkJntGYjXPDw+PWisKKSsAeZFTskvg@mail.gmail.com Assertion failure when streaming logical changes] (crash in walsender replaying from a logical decoding slot)<br />
* [http://www.postgresql.org/message-id/20141128205453.GA1737@alvh.no-ip.org no test programs in contrib] fix src/test/modules to work on MSVC<br />
* [http://www.postgresql.org/message-id/CAG6W84JA8bhrEzDvv6UaTOyZGBPwDnQb7ZqJRm6wtJdn+mBY9Q@mail.gmail.com Improve GB18030 <-> UTF8 encoding conversions]<br />
* [http://www.postgresql.org/message-id/20150312.213812.115476889.horiguchi.kyotaro@lab.ntt.co.jp alter user/role CURRENT_USER] CURRENT_USER needs some fixes<br />
* [http://www.postgresql.org/message-id/55427924.9090806@dunslane.net transforms vs CLOBBER_CACHE_ALWAYS]<br />
<br />
[[Category:PostgreSQL_9.5]]</div>Kaigaihttps://wiki.postgresql.org/index.php?title=CustomScanInterface&diff=24241CustomScanInterface2015-02-13T07:47:52Z<p>Kaigai: /* CustomScan structure */</p>
<hr />
<div>= Writing A Custom Scan Provider =<br />
<br />
== Overview ==<br />
<br />
Prior to query execution, the PostgreSQL planner constructs a plan tree that usually consists of built-in plan nodes (IE: SeqScan, HashJoin, etc).<br />
The custom-scan interface allows extensions to create a custom-scan provider that implements its own logic, in addition to the built-in nodes, for scanning relations.<br />
If a custom-scan node is chosen by the planner, callback functions associated with this custom-scan node shall be invoked during query execution. The custom-scan provider is responsible for returning equivalent result set as built-in logic would, but it is free to scan the relation according to its own logic.<br />
<br />
This chapter explains how to write a custom-scan provider.<br />
<br />
== Interaction with Planner ==<br />
<br />
Planner queries extension whether it can provide alternative scan path using a hook; set_rel_pathlist_hook for relation scan, during a plan construction.<br />
This invocation gives a couple of information for extension to determine whether it can offer alternative scan path on the particular relation, then it can arbitrarily add custom-path on RelOptInfo of the relation to be scanned.<br />
Custom-scan provider is responsible to set reasonable cost estimation on the CustomPath node; that is the only way for the core planner to determine which Path shall be chosen. The logic how built-in scan-path may be a good example.<br />
<br />
Once a custom-path got chosen by planner, custom-scan provider has to populate a CustomScan structure according to the custom-path. The CustomScan structure has two special fields to keep private information; custom_exprs and custom_private. The custom_exprs is intended to save a couple of expression trees that shall be updated on setrefs.c and subselect.c. custom_private is intended to save private information nobody will touch except for the custom-scan provider itself.<br />
A plan-tree, which contains custom-scan node, can be duplicated using copyObject(), so anything stored in these two fields must support copyObject().<br />
<br />
== Interaction with Executor ==<br />
<br />
Once a plan-tree is moved to the executor, it constructs plan-state objects according to the plan-node. Custom-scan is not an exception. Executor invokes a callback to populate CustomScanState node, if it found CustomScan node in the supplied plan-tree. Unlike CustomScan node, it does not have fields to save private information. Instead of these fields, custom-scan provider can allocate larger object than CustomScanState even though its header layout is compatible with CustomScanState. It looks like a relationship of ScanState structure towards PlanState; that expands scan specific fields towards generic plan-state. In addition, custom-scan provider can expand fields on demand.<br />
<br />
Once a CustomScanState gets constructed, BeginCustomScan is invoked during executor initialization; ExecCustomScan is repeatedly called during execution (returning a TupleTableSlot with each fetched record), then EndCustomScan is invoked on cleanup of the executor.<br />
<br />
== Hooks to add custom-paths ==<br />
<br />
Here is two hooks for extensions to add custom-paths, according to the expected tasks.<br />
<br />
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
Index rti,<br />
RangeTblEntry *rte);<br />
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;<br />
<br />
This hook is invoked when the planner investigates the potential scan paths on a particular relation.<br />
The custom-scan provider can add custom-path on the supplied relation if it can offer alternative scan paths, using add_paths().<br />
<br />
typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *joinrel,<br />
RelOptInfo *outerrel,<br />
RelOptInfo *innerrel,<br />
List *restrictlist,<br />
JoinType jointype,<br />
SpecialJoinInfo *sjinfo,<br />
SemiAntiJoinFactors *semifactors,<br />
Relids param_source_rels,<br />
Relids extra_lateral_rels);<br />
extern PGDLLIMPORT set_join_pathlist_hook_type set_join_pathlist_hook;<br />
<br />
This hook is invoked when the planner investigates potential join paths on a particular relations join.<br />
The custom-scan provider can add custom-path on the supplied join-relation if it can offer alternative scan path, using add_paths().<br />
From the viewpoint of the executor, it looks to a relation scan but on a pseudo relation that is materialized from the multiple relations. Custom-scan provider is expected to process relations join with its own logic internally, then return tuple records according to the tuple-descriptor of this scan node.<br />
<br />
== Functions of CustomPathMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node. The core backend invokes these callbacks during query planning.<br />
<br />
Plan *(*PlanCustomPath) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
CustomPath *best_path,<br />
List *tlist,<br />
List *clauses);<br />
<br />
This callback is invoked when the core backend tries to populate CustomScan node according to the supplied CustomPath node. The custom-scan provider is responsible to allocate a CustomScan node and initialize each fields of them.<br />
<br />
<br />
void (*TextOutCustomPath) (StringInfo str,<br />
const CustomPath *node);<br />
<br />
This optional callback will be invoked when nodeToString() tries to create a text representation of CustomPath node. A custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_private shall be transformed to text representation by the core, so nothing to do by extension.<br />
<br />
== CustomScan structure ==<br />
<br />
PlanCustomPath method will construct a custom-plan object, if planner considered the path is enough reasonable.<br />
CustomScan is the only custom plan node type that we can populate from the custom-path node at this moment.<br />
It has a few key fields to control the behavior, to be set and initialized by custom-scan provider.<br />
<br />
'''scanrelid''' is a sub-field of Scan structure within CustomScan. Usually, it is a positive index number of range-table entries to clarify which relation is associated with this scan node. However, CustomScan (and ForeignScan) node has special meaning if scanrelid is 0. In this case, CustomScan node is not associated with a certain physical relation, therefore, it performs a scan on pseudo-relation.<br />
<br />
'''flags''' gives the planner hint about expected behavior of this custom-scan node. '''CUSTOMPATH_SUPPORT_BACKWARD_SCAN''' introduce the custom-scan node supports backward scan. '''CUSTOMPATH_SUPPORT_MARK_RESTORE''' introduce the custom-scan node supports mark and restore position during execution.<br />
<br />
'''custom_ps_tlist''' is a list of TargetEntry to inform the core planner/executor expected data type of the records to be returned. Executor initializes CustomScanState node according to the pseudo-scan tlist, even if it is not associated with a physical relation.<br />
<br />
== Functions of CustomScanMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node, then the core backend invokes these callbacks during query planning.<br />
<br />
Node *(*CreateCustomScanState) (CustomScan *cscan);<br />
<br />
This callback shall be invoked when the core backend tries to populate CustomScanState node according to the supplied CustomScan node. The custom-scan provider is responsible to allocate a CustomScanState (or its own data-type enhanced from it), but no need to initialize the fields here, because ExecInitCustomScan initializes the fields in CustomScanState, then BeginCustomScan shall be kicked on the end of executor initialization.<br />
<br />
void (*TextOutCustomScan) (StringInfo str,<br />
const CustomScan *node);<br />
<br />
This optional callback shall be invoked when nodeToString() tries to make text representation of CustomScan node. Custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_exprs and custom_private are transformed to text representation by the core and it is not allowed to expand the data-type by extension, thus, we usually don't need to implement this callback.<br />
<br />
== Functions of CustomExecMethods ==<br />
<br />
void (*BeginCustomScan) (CustomScanState *node,<br />
EState *estate,<br />
int eflags);<br />
<br />
This callback allows a custom-scan provider to initialize the CustomScanState node.<br />
The supplied CustomScanState is partially initialized according to the scanrelid of CustomScan node. If the custom-scan provider wants to apply additional initialization to the private fields, it can be done by this callback.<br />
<br />
TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to produce the next tuple of the relation scan.<br />
If any tuples, it should set it on the ps_resultxxx then returns the tuple-slot. Elsewhere, NULL or empty slot should be returned to inform the upper node end of relation scan.<br />
<br />
void (*EndCustomScan) (CustomScanState *node);<br />
<br />
This callback allows a custom-scan provider to cleanup the CustomScanState node.<br />
If it holds any private (and not released automatically) resources on the supplied CustomScanState, it can release these resources prior to the cleanup of the common portion.<br />
<br />
void (*ReScanCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to rewind the current scan position to the head of relation. Custom-scan provider is expected to reset its internal state to restart the relation scan again.<br />
<br />
void (*MarkPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to save the current scan position on its internal state. It shall be able to restore the position using RestrPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag.<br />
<br />
void (*RestrPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to restore the previous scan position that was saved by MarkPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag .<br />
<br />
void (*ExplainCustomScan) (CustomScanState *node,<br />
List *ancestors,<br />
ExplainState *es);<br />
<br />
This optional callback allows custom-scan provider to output additional information on EXPLAIN command that involves custom-scan node. Note that it can output common items; target-list, qualifiers, relation to be scanned. So, it can be used when custom-scan provider wants to show something others in addition to the items.</div>Kaigaihttps://wiki.postgresql.org/index.php?title=CustomScanInterface&diff=24240CustomScanInterface2015-02-13T07:41:52Z<p>Kaigai: /* Functions of CustomScanMethods */</p>
<hr />
<div>= Writing A Custom Scan Provider =<br />
<br />
== Overview ==<br />
<br />
Prior to query execution, the PostgreSQL planner constructs a plan tree that usually consists of built-in plan nodes (IE: SeqScan, HashJoin, etc).<br />
The custom-scan interface allows extensions to create a custom-scan provider that implements its own logic, in addition to the built-in nodes, for scanning relations.<br />
If a custom-scan node is chosen by the planner, callback functions associated with this custom-scan node shall be invoked during query execution. The custom-scan provider is responsible for returning equivalent result set as built-in logic would, but it is free to scan the relation according to its own logic.<br />
<br />
This chapter explains how to write a custom-scan provider.<br />
<br />
== Interaction with Planner ==<br />
<br />
Planner queries extension whether it can provide alternative scan path using a hook; set_rel_pathlist_hook for relation scan, during a plan construction.<br />
This invocation gives a couple of information for extension to determine whether it can offer alternative scan path on the particular relation, then it can arbitrarily add custom-path on RelOptInfo of the relation to be scanned.<br />
Custom-scan provider is responsible to set reasonable cost estimation on the CustomPath node; that is the only way for the core planner to determine which Path shall be chosen. The logic how built-in scan-path may be a good example.<br />
<br />
Once a custom-path got chosen by planner, custom-scan provider has to populate a CustomScan structure according to the custom-path. The CustomScan structure has two special fields to keep private information; custom_exprs and custom_private. The custom_exprs is intended to save a couple of expression trees that shall be updated on setrefs.c and subselect.c. custom_private is intended to save private information nobody will touch except for the custom-scan provider itself.<br />
A plan-tree, which contains custom-scan node, can be duplicated using copyObject(), so anything stored in these two fields must support copyObject().<br />
<br />
== Interaction with Executor ==<br />
<br />
Once a plan-tree is moved to the executor, it constructs plan-state objects according to the plan-node. Custom-scan is not an exception. Executor invokes a callback to populate CustomScanState node, if it found CustomScan node in the supplied plan-tree. Unlike CustomScan node, it does not have fields to save private information. Instead of these fields, custom-scan provider can allocate larger object than CustomScanState even though its header layout is compatible with CustomScanState. It looks like a relationship of ScanState structure towards PlanState; that expands scan specific fields towards generic plan-state. In addition, custom-scan provider can expand fields on demand.<br />
<br />
Once a CustomScanState gets constructed, BeginCustomScan is invoked during executor initialization; ExecCustomScan is repeatedly called during execution (returning a TupleTableSlot with each fetched record), then EndCustomScan is invoked on cleanup of the executor.<br />
<br />
== Hooks to add custom-paths ==<br />
<br />
Here is two hooks for extensions to add custom-paths, according to the expected tasks.<br />
<br />
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
Index rti,<br />
RangeTblEntry *rte);<br />
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;<br />
<br />
This hook is invoked when the planner investigates the potential scan paths on a particular relation.<br />
The custom-scan provider can add custom-path on the supplied relation if it can offer alternative scan paths, using add_paths().<br />
<br />
typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *joinrel,<br />
RelOptInfo *outerrel,<br />
RelOptInfo *innerrel,<br />
List *restrictlist,<br />
JoinType jointype,<br />
SpecialJoinInfo *sjinfo,<br />
SemiAntiJoinFactors *semifactors,<br />
Relids param_source_rels,<br />
Relids extra_lateral_rels);<br />
extern PGDLLIMPORT set_join_pathlist_hook_type set_join_pathlist_hook;<br />
<br />
This hook is invoked when the planner investigates potential join paths on a particular relations join.<br />
The custom-scan provider can add custom-path on the supplied join-relation if it can offer alternative scan path, using add_paths().<br />
From the viewpoint of the executor, it looks to a relation scan but on a pseudo relation that is materialized from the multiple relations. Custom-scan provider is expected to process relations join with its own logic internally, then return tuple records according to the tuple-descriptor of this scan node.<br />
<br />
== Functions of CustomPathMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node. The core backend invokes these callbacks during query planning.<br />
<br />
Plan *(*PlanCustomPath) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
CustomPath *best_path,<br />
List *tlist,<br />
List *clauses);<br />
<br />
This callback is invoked when the core backend tries to populate CustomScan node according to the supplied CustomPath node. The custom-scan provider is responsible to allocate a CustomScan node and initialize each fields of them.<br />
<br />
<br />
void (*TextOutCustomPath) (StringInfo str,<br />
const CustomPath *node);<br />
<br />
This optional callback will be invoked when nodeToString() tries to create a text representation of CustomPath node. A custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_private shall be transformed to text representation by the core, so nothing to do by extension.<br />
<br />
== CustomScan structure ==<br />
<br />
PlanCustomPath method will construct a custom-plan object, if planner considered the path is enough reasonable.<br />
CustomScan is the only custom plan node type that we can populate from the custom-path node at this moment.<br />
It has a few key fields to control the behavior, to be set and initialized by custom-scan provider.<br />
<br />
'''scanrelid''' is a sub-field of Scan structure within CustomScan. Usually, it is a positive index number of range-table entries to clarify which relation is associated with this scan node. However, CustomScan (and ForeignScan) node has special meaning if scanrelid is 0. In this case, CustomScan node is not associated with a certain physical relation, therefore, it performs a scan on pseudo-relation.<br />
<br />
'''ps_tlist''' is a list of TargetEntry to inform the core planner/executor expected data type of the records to be returned. Executor initializes CustomScanState node according to the pseudo-scan tlist, even if it is not associated with a physical relation.<br />
<br />
<br />
== Functions of CustomScanMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node, then the core backend invokes these callbacks during query planning.<br />
<br />
Node *(*CreateCustomScanState) (CustomScan *cscan);<br />
<br />
This callback shall be invoked when the core backend tries to populate CustomScanState node according to the supplied CustomScan node. The custom-scan provider is responsible to allocate a CustomScanState (or its own data-type enhanced from it), but no need to initialize the fields here, because ExecInitCustomScan initializes the fields in CustomScanState, then BeginCustomScan shall be kicked on the end of executor initialization.<br />
<br />
void (*TextOutCustomScan) (StringInfo str,<br />
const CustomScan *node);<br />
<br />
This optional callback shall be invoked when nodeToString() tries to make text representation of CustomScan node. Custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_exprs and custom_private are transformed to text representation by the core and it is not allowed to expand the data-type by extension, thus, we usually don't need to implement this callback.<br />
<br />
== Functions of CustomExecMethods ==<br />
<br />
void (*BeginCustomScan) (CustomScanState *node,<br />
EState *estate,<br />
int eflags);<br />
<br />
This callback allows a custom-scan provider to initialize the CustomScanState node.<br />
The supplied CustomScanState is partially initialized according to the scanrelid of CustomScan node. If the custom-scan provider wants to apply additional initialization to the private fields, it can be done by this callback.<br />
<br />
TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to produce the next tuple of the relation scan.<br />
If any tuples, it should set it on the ps_resultxxx then returns the tuple-slot. Elsewhere, NULL or empty slot should be returned to inform the upper node end of relation scan.<br />
<br />
void (*EndCustomScan) (CustomScanState *node);<br />
<br />
This callback allows a custom-scan provider to cleanup the CustomScanState node.<br />
If it holds any private (and not released automatically) resources on the supplied CustomScanState, it can release these resources prior to the cleanup of the common portion.<br />
<br />
void (*ReScanCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to rewind the current scan position to the head of relation. Custom-scan provider is expected to reset its internal state to restart the relation scan again.<br />
<br />
void (*MarkPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to save the current scan position on its internal state. It shall be able to restore the position using RestrPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag.<br />
<br />
void (*RestrPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to restore the previous scan position that was saved by MarkPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag .<br />
<br />
void (*ExplainCustomScan) (CustomScanState *node,<br />
List *ancestors,<br />
ExplainState *es);<br />
<br />
This optional callback allows custom-scan provider to output additional information on EXPLAIN command that involves custom-scan node. Note that it can output common items; target-list, qualifiers, relation to be scanned. So, it can be used when custom-scan provider wants to show something others in addition to the items.</div>Kaigaihttps://wiki.postgresql.org/index.php?title=CustomScanInterface&diff=24239CustomScanInterface2015-02-13T06:38:29Z<p>Kaigai: /* Hooks to add custom-paths */</p>
<hr />
<div>= Writing A Custom Scan Provider =<br />
<br />
== Overview ==<br />
<br />
Prior to query execution, the PostgreSQL planner constructs a plan tree that usually consists of built-in plan nodes (IE: SeqScan, HashJoin, etc).<br />
The custom-scan interface allows extensions to create a custom-scan provider that implements its own logic, in addition to the built-in nodes, for scanning relations.<br />
If a custom-scan node is chosen by the planner, callback functions associated with this custom-scan node shall be invoked during query execution. The custom-scan provider is responsible for returning equivalent result set as built-in logic would, but it is free to scan the relation according to its own logic.<br />
<br />
This chapter explains how to write a custom-scan provider.<br />
<br />
== Interaction with Planner ==<br />
<br />
Planner queries extension whether it can provide alternative scan path using a hook; set_rel_pathlist_hook for relation scan, during a plan construction.<br />
This invocation gives a couple of information for extension to determine whether it can offer alternative scan path on the particular relation, then it can arbitrarily add custom-path on RelOptInfo of the relation to be scanned.<br />
Custom-scan provider is responsible to set reasonable cost estimation on the CustomPath node; that is the only way for the core planner to determine which Path shall be chosen. The logic how built-in scan-path may be a good example.<br />
<br />
Once a custom-path got chosen by planner, custom-scan provider has to populate a CustomScan structure according to the custom-path. The CustomScan structure has two special fields to keep private information; custom_exprs and custom_private. The custom_exprs is intended to save a couple of expression trees that shall be updated on setrefs.c and subselect.c. custom_private is intended to save private information nobody will touch except for the custom-scan provider itself.<br />
A plan-tree, which contains custom-scan node, can be duplicated using copyObject(), so anything stored in these two fields must support copyObject().<br />
<br />
== Interaction with Executor ==<br />
<br />
Once a plan-tree is moved to the executor, it constructs plan-state objects according to the plan-node. Custom-scan is not an exception. Executor invokes a callback to populate CustomScanState node, if it found CustomScan node in the supplied plan-tree. Unlike CustomScan node, it does not have fields to save private information. Instead of these fields, custom-scan provider can allocate larger object than CustomScanState even though its header layout is compatible with CustomScanState. It looks like a relationship of ScanState structure towards PlanState; that expands scan specific fields towards generic plan-state. In addition, custom-scan provider can expand fields on demand.<br />
<br />
Once a CustomScanState gets constructed, BeginCustomScan is invoked during executor initialization; ExecCustomScan is repeatedly called during execution (returning a TupleTableSlot with each fetched record), then EndCustomScan is invoked on cleanup of the executor.<br />
<br />
== Hooks to add custom-paths ==<br />
<br />
Here is two hooks for extensions to add custom-paths, according to the expected tasks.<br />
<br />
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
Index rti,<br />
RangeTblEntry *rte);<br />
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;<br />
<br />
This hook is invoked when the planner investigates the potential scan paths on a particular relation.<br />
The custom-scan provider can add custom-path on the supplied relation if it can offer alternative scan paths, using add_paths().<br />
<br />
typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *joinrel,<br />
RelOptInfo *outerrel,<br />
RelOptInfo *innerrel,<br />
List *restrictlist,<br />
JoinType jointype,<br />
SpecialJoinInfo *sjinfo,<br />
SemiAntiJoinFactors *semifactors,<br />
Relids param_source_rels,<br />
Relids extra_lateral_rels);<br />
extern PGDLLIMPORT set_join_pathlist_hook_type set_join_pathlist_hook;<br />
<br />
This hook is invoked when the planner investigates potential join paths on a particular relations join.<br />
The custom-scan provider can add custom-path on the supplied join-relation if it can offer alternative scan path, using add_paths().<br />
From the viewpoint of the executor, it looks to a relation scan but on a pseudo relation that is materialized from the multiple relations. Custom-scan provider is expected to process relations join with its own logic internally, then return tuple records according to the tuple-descriptor of this scan node.<br />
<br />
== Functions of CustomPathMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node. The core backend invokes these callbacks during query planning.<br />
<br />
Plan *(*PlanCustomPath) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
CustomPath *best_path,<br />
List *tlist,<br />
List *clauses);<br />
<br />
This callback is invoked when the core backend tries to populate CustomScan node according to the supplied CustomPath node. The custom-scan provider is responsible to allocate a CustomScan node and initialize each fields of them.<br />
<br />
<br />
void (*TextOutCustomPath) (StringInfo str,<br />
const CustomPath *node);<br />
<br />
This optional callback will be invoked when nodeToString() tries to create a text representation of CustomPath node. A custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_private shall be transformed to text representation by the core, so nothing to do by extension.<br />
<br />
== Functions of CustomScanMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node, then the core backend invokes these callbacks during query planning.<br />
<br />
Node *(*CreateCustomScanState) (CustomScan *cscan);<br />
<br />
This callback shall be invoked when the core backend tries to populate CustomScanState node according to the supplied CustomScan node. The custom-scan provider is responsible to allocate a CustomScanState (or its own data-type enhanced from it), but no need to initialize the fields here, because ExecInitCustomScan initializes the fields in CustomScanState, then BeginCustomScan shall be kicked on the end of executor initialization.<br />
<br />
void (*TextOutCustomScan) (StringInfo str,<br />
const CustomScan *node);<br />
<br />
This optional callback shall be invoked when nodeToString() tries to make text representation of CustomScan node. Custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_exprs and custom_private are transformed to text representation by the core and it is not allowed to expand the data-type by extension, thus, we usually don't need to implement this callback.<br />
<br />
== Functions of CustomExecMethods ==<br />
<br />
void (*BeginCustomScan) (CustomScanState *node,<br />
EState *estate,<br />
int eflags);<br />
<br />
This callback allows a custom-scan provider to initialize the CustomScanState node.<br />
The supplied CustomScanState is partially initialized according to the scanrelid of CustomScan node. If the custom-scan provider wants to apply additional initialization to the private fields, it can be done by this callback.<br />
<br />
TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to produce the next tuple of the relation scan.<br />
If any tuples, it should set it on the ps_resultxxx then returns the tuple-slot. Elsewhere, NULL or empty slot should be returned to inform the upper node end of relation scan.<br />
<br />
void (*EndCustomScan) (CustomScanState *node);<br />
<br />
This callback allows a custom-scan provider to cleanup the CustomScanState node.<br />
If it holds any private (and not released automatically) resources on the supplied CustomScanState, it can release these resources prior to the cleanup of the common portion.<br />
<br />
void (*ReScanCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to rewind the current scan position to the head of relation. Custom-scan provider is expected to reset its internal state to restart the relation scan again.<br />
<br />
void (*MarkPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to save the current scan position on its internal state. It shall be able to restore the position using RestrPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag.<br />
<br />
void (*RestrPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to restore the previous scan position that was saved by MarkPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag .<br />
<br />
void (*ExplainCustomScan) (CustomScanState *node,<br />
List *ancestors,<br />
ExplainState *es);<br />
<br />
This optional callback allows custom-scan provider to output additional information on EXPLAIN command that involves custom-scan node. Note that it can output common items; target-list, qualifiers, relation to be scanned. So, it can be used when custom-scan provider wants to show something others in addition to the items.</div>Kaigaihttps://wiki.postgresql.org/index.php?title=CustomScanInterface&diff=23921CustomScanInterface2014-12-02T14:11:02Z<p>Kaigai: </p>
<hr />
<div>= Writing A Custom Scan Provider =<br />
<br />
== Ovewview ==<br />
<br />
Prior to query execution, the planner of PostgreSQL constructs a plan tree that is usually consists of built-in plan node (like SeqScan, HashJoin, ...).<br />
Custom-scan interface allows extensions to perform as a custom-scan provider that implement its own logic, in addition to the built-in ones, to scan relations.<br />
Once a custom-scan node can offer the cheapest cost estimation for relation scan, thus, chosen by the planner, callback functions associated with this custom-scan node shall be invoked during query execution. Even though custom-scan provider is responsible to return equivalent result set as built-in logic doing, it can scan the relation according to its own logic.<br />
This chapter introduce the way to write a custom-scan provider.<br />
<br />
== Interaction with Planner ==<br />
<br />
Planner queries extension whether it can provide alternative scan path using a hook; set_rel_pathlist_hook for relation scan, during a plan construction.<br />
This invocation gives a couple of information for extension to determine whether it can offer alternative scan path on the particular relation, then it can arbitrarily add custom-path on RelOptInfo of the relation to be scanned.<br />
Custom-scan provider is responsible to set reasonable cost estimation on the CustomPath node; that is the only way for the core planner to determine which Path shall be chosen. The logic how built-in scan-path may be a good example.<br />
<br />
Once a custom-path got chosen by planner, custom-scan provider has to populate a CustomScan structure according to the custom-path. The CustomScan structure has two special fields to keep private information; custom_exprs and custom_private. The custom_exprs is intended to save a couple of expression trees that shall be updated on setrefs.c and subselect.c. Onther other hand, as literal, custom_private is intended to save private information nobody will touch except for the custom-scan provider itself.<br />
A plan-tree, which contains custom-scan node, can be duplicated using copyObject(), so these two fields have to be safe to copyObject().<br />
<br />
== Interaction with Executor ==<br />
<br />
Once a plan-tree is moved to the executor, it constructs plan-state objects according to the plan-node. Custom-scan is not an exception. Executor invokes a callback to populate CustomScanState node, if it found CustomScan node in the supplied plan-tree. Unlike CustomScan node, it does not have fields to save private information. Instead of these fields, custom-scan provider can allocate larger object than CustomScanState even though its header layout is compatible with CustomScanState. It looks like a relationship of ScanState structure towards PlanState; that expands scan specific fields towards generic plan-state. In addition, custom-scan provider can expand fields on demand.<br />
<br />
Once a CustomScanState gets constructed, BeginCustomScan is invoked on initialization of the executor, ExecCustomScan is repeatedly involed on execution of the relation scan to get TupleTableSlot with fetched record, then EndCustomScan is invoked on cleanup of the executor.<br />
<br />
== Hooks to add custom-paths ==<br />
<br />
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
Index rti,<br />
RangeTblEntry *rte);<br />
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;<br />
<br />
This hooks is invoked when planner investigate all the potential scan path on a particular relation.<br />
Then, custom-scan provider can add custom-path on the supplied relation if it can offer alternative scan paths, using add_paths().<br />
<br />
== Functions of CustomPathMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node, then the core backend invokes these callbacks during query planning.<br />
<br />
Plan *(*PlanCustomPath) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
CustomPath *best_path,<br />
List *tlist,<br />
List *clauses);<br />
<br />
This callback shall be invoked when the core backend tries to populate CustomScan node according to the supplied CustomPath node. The custom-scan provider is responsible to allocate a CustomScan node and initialize each fields of them.<br />
<br />
<br />
void (*TextOutCustomPath) (StringInfo str,<br />
const CustomPath *node);<br />
<br />
This optional callback shall be invoked when nodeToString() tries to make text representation of CustomPath node. Custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_private shall be transformed to text representation by the core, so nothing to do by extension.<br />
<br />
== Functions of CustomScanMethods ==<br />
<br />
A CustomPathMethods table contains a set of callbacks related to CustomPath node, then the core backend invokes these callbacks during query planning.<br />
<br />
Node *(*CreateCustomScanState) (CustomScan *cscan);<br />
<br />
This callback shall be invoked when the core backend tries to populate CustomScanState node according to the supplied CustomScan node. The custom-scan provider is responsible to allocate a CustomScanState (or its own data-type enhanced from it), but no need to initialize the fields here, because ExecInitCustomScan initializes the fields in CustomScanState, then BeginCustomScan shall be kicked on the end of executor initialization.<br />
<br />
void (*TextOutCustomScan) (StringInfo str,<br />
const CustomScan *node);<br />
<br />
This optional callback shall be invoked when nodeToString() tries to make text representation of CustomScan node. Custom-scan provider can utilize this callback, if it wants to output something additional. Note that expression nodes linked to custom_exprs and custom_private are transformed to text representation by the core and it is not allowed to expand the data-type by extension, thus, we usually don't need to implement this callback.<br />
<br />
== Functions of CustomExecMethods ==<br />
<br />
void (*BeginCustomScan) (CustomScanState *node,<br />
EState *estate,<br />
int eflags);<br />
<br />
This callback allows custom-scan provider to initializes the CustomScanState node.<br />
The supplied CustomScanState is partially initialized according to the scanrelid of CustomScan node. If custom-scan provider wants to put additional initialization on the private fields, it can be implemented on this callback.<br />
<br />
TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to produce the next tuple of the relation scan.<br />
If any tuples, it should set it on the ps_resultxxx then returns the tuple-slot. Elsewhere, NULL or empty slot should be returned to inform the upper node end of relation scan.<br />
<br />
void (*EndCustomScan) (CustomScanState *node);<br />
<br />
This callback allows custom-scan provider to cleanup the CustomScanState node.<br />
If it hold any private (and not released automatically) resources on the supplied CustomScanState, it can release these resources prior to the cleanup of common portion.<br />
<br />
void (*ReScanCustomScan) (CustomScanState *node);<br />
<br />
This callback requires custom-scan provider to rewind the current scan position to the head of relation. Custom-scan provider is expected to reset its internal state to restart the relation scan again.<br />
<br />
void (*MarkPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to save the current scan position on its internal state. It shall be able to restore the position using RestrPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag.<br />
<br />
void (*RestrPosCustomScan) (CustomScanState *node);<br />
<br />
This optional callback requires custom-scan provider to restore the previous scan position that was saved by MarkPosCustomScan callback.<br />
It shall be never called without CUSTOMPATH_SUPPORT_MARK_RESTORE flag .<br />
<br />
void (*ExplainCustomScan) (CustomScanState *node,<br />
List *ancestors,<br />
ExplainState *es);<br />
<br />
This optional callback allows custom-scan provider to output additional information on EXPLAIN command that involves custom-scan node. Note that it can output common items; target-list, qualifiers, relation to be scanned. So, it can be used when custom-scan provider wants to show something others in addition to the items.</div>Kaigaihttps://wiki.postgresql.org/index.php?title=CustomScanInterface&diff=23882CustomScanInterface2014-12-01T05:47:44Z<p>Kaigai: </p>
<hr />
<div>= Writing A Custom Scan Provider =<br />
<br />
== Ovewview ==<br />
<br />
Prior to query execution, the planner of PostgreSQL constructs a plan tree that is usually consists of built-in plan node (like SeqScan, HashJoin, ...).<br />
Custom-scan interface allows extensions to perform as a custom-scan provider that implement its own logic, in addition to the built-in ones, to scan relations.<br />
Once a custom-scan node can offer the cheapest cost estimation for relation scan, thus, chosen by the planner, callback functions associated with this custom-scan node shall be invoked during query execution. Even though custom-scan provider is responsible to return equivalent result set as built-in logic doing, it can scan the relation according to its own logic.<br />
This chapter introduce the way to write a custom-scan provider.<br />
<br />
== Interaction with Planner ==<br />
<br />
Planner queries extension whether it can provide alternative scan path using a hook; set_rel_pathlist_hook for relation scan, during a plan construction.<br />
This invocation gives a couple of information for extension to determine whether it can offer alternative scan path on the particular relation, then it can arbitrarily add custom-path on RelOptInfo of the relation to be scanned.<br />
Custom-scan provider is responsible to set reasonable cost estimation on the CustomPath node; that is the only way for the core planner to determine which Path shall be chosen. The logic how built-in scan-path may be a good example.<br />
<br />
Once a custom-path got chosen by planner, custom-scan provider has to populate a CustomScan structure according to the custom-path. The CustomScan structure has two special fields to keep private information; custom_exprs and custom_private. The custom_exprs is intended to save a couple of expression trees that shall be updated on setrefs.c and subselect.c. Onther other hand, as literal, custom_private is intended to save private information nobody will touch except for the custom-scan provider itself.<br />
A plan-tree, which contains custom-scan node, can be duplicated using copyObject(), so these two fields have to be safe to copyObject().<br />
<br />
== Interaction with Executor ==<br />
<br />
Once a plan-tree is moved to the executor, it constructs plan-state objects according to the plan-node. Custom-scan is not an exception. Executor invokes a callback to populate CustomScanState node, if it found CustomScan node in the supplied plan-tree. Unlike CustomScan node, it does not have fields to save private information. Instead of these fields, custom-scan provider can allocate larger object than CustomScanState even though its header layout is compatible with CustomScanState. It looks like a relationship of ScanState structure towards PlanState; that expands scan specific fields towards generic plan-state. In addition, custom-scan provider can expand fields on demand.<br />
<br />
Once a CustomScanState gets constructed, BeginCustomScan is invoked on initialization of the executor, ExecCustomScan is repeatedly involed on execution of the relation scan to get TupleTableSlot with fetched record, then EndCustomScan is invoked on cleanup of the executor.<br />
<br />
== Hooks to add custom-paths ==<br />
<br />
<br />
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
Index rti,<br />
RangeTblEntry *rte);<br />
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;<br />
<br />
<br />
<br />
<br />
<br />
<br />
== Functions of CustomPathMethods ==<br />
<br />
Plan *(*PlanCustomPath) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
CustomPath *best_path,<br />
List *tlist,<br />
List *clauses);<br />
<br />
void (*TextOutCustomPath) (StringInfo str,<br />
const CustomPath *node);<br />
<br />
<br />
== Functions of CustomScanMethods ==<br />
<br />
Node *(*CreateCustomScanState) (CustomScan *cscan);<br />
<br />
void (*TextOutCustomScan) (StringInfo str,<br />
const CustomScan *node);<br />
<br />
<br />
<br />
== Functions of CustomExecMethods ==<br />
<br />
void (*BeginCustomScan) (CustomScanState *node,<br />
EState *estate,<br />
int eflags);<br />
<br />
TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);<br />
<br />
void (*EndCustomScan) (CustomScanState *node);<br />
<br />
void (*ReScanCustomScan) (CustomScanState *node);<br />
<br />
void (*MarkPosCustomScan) (CustomScanState *node);<br />
<br />
void (*RestrPosCustomScan) (CustomScanState *node);<br />
<br />
void (*ExplainCustomScan) (CustomScanState *node,<br />
List *ancestors,<br />
ExplainState *es);</div>Kaigaihttps://wiki.postgresql.org/index.php?title=CustomScanInterface&diff=23881CustomScanInterface2014-12-01T05:47:04Z<p>Kaigai: Created page with "= Writing A Custom Scan Provider = == Ovewview == Prior to query execution, the planner of PostgreSQL constructs a plan tree that is usually consists of built-in plan node (..."</p>
<hr />
<div>= Writing A Custom Scan Provider =<br />
<br />
== Ovewview ==<br />
<br />
Prior to query execution, the planner of PostgreSQL constructs a plan tree that is usually consists of built-in plan node (like SeqScan, HashJoin, ...).<br />
Custom-scan interface allows extensions to perform as a custom-scan provider that implement its own logic, in addition to the built-in ones, to scan relations.<br />
Once a custom-scan node can offer the cheapest cost estimation for relation scan, thus, chosen by the planner, callback functions associated with this custom-scan node shall be invoked during query execution. Even though custom-scan provider is responsible to return equivalent result set as built-in logic doing, it can scan the relation according to its own logic.<br />
This chapter introduce the way to write a custom-scan provider.<br />
<br />
== Interaction with Planner ==<br />
<br />
Planner queries extension whether it can provide alternative scan path using a hook; set_rel_pathlist_hook for relation scan, during a plan construction.<br />
This invocation gives a couple of information for extension to determine whether it can offer alternative scan path on the particular relation, then it can arbitrarily add custom-path on RelOptInfo of the relation to be scanned.<br />
Custom-scan provider is responsible to set reasonable cost estimation on the CustomPath node; that is the only way for the core planner to determine which Path shall be chosen. The logic how built-in scan-path may be a good example.<br />
<br />
Once a custom-path got chosen by planner, custom-scan provider has to populate a CustomScan structure according to the custom-path. The CustomScan structure has two special fields to keep private information; custom_exprs and custom_private. The custom_exprs is intended to save a couple of expression trees that shall be updated on setrefs.c and subselect.c. Onther other hand, as literal, custom_private is intended to save private information nobody will touch except for the custom-scan provider itself.<br />
A plan-tree, which contains custom-scan node, can be duplicated using copyObject(), so these two fields have to be safe to copyObject().<br />
<br />
== Interaction with Executor ==<br />
<br />
Once a plan-tree is moved to the executor, it constructs plan-state objects according to the plan-node. Custom-scan is not an exception. Executor invokes a callback to populate CustomScanState node, if it found CustomScan node in the supplied plan-tree. Unlike CustomScan node, it does not have fields to save private information. Instead of these fields, custom-scan provider can allocate larger object than CustomScanState even though its header layout is compatible with CustomScanState. It looks like a relationship of ScanState structure towards PlanState; that expands scan specific fields towards generic plan-state. In addition, custom-scan provider can expand fields on demand.<br />
<br />
Once a CustomScanState gets constructed, BeginCustomScan is invoked on initialization of the executor, ExecCustomScan is repeatedly involed on execution of the relation scan to get TupleTableSlot with fetched record, then EndCustomScan is invoked on cleanup of the executor.<br />
<br />
== Hooks to add custom-paths ==<br />
<br />
<br />
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
Index rti,<br />
RangeTblEntry *rte);<br />
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;<br />
<br />
<br />
<br />
<br />
<br />
<br />
== Functions of CustomPathMethods ==<br />
<br />
Plan *(*PlanCustomPath) (PlannerInfo *root,<br />
RelOptInfo *rel,<br />
CustomPath *best_path,<br />
List *tlist,<br />
List *clauses);<br />
<br />
<br />
void (*TextOutCustomPath) (StringInfo str,<br />
const CustomPath *node);<br />
<br />
<br />
== Functions of CustomScanMethods ==<br />
<br />
Node *(*CreateCustomScanState) (CustomScan *cscan);<br />
<br />
void (*TextOutCustomScan) (StringInfo str,<br />
const CustomScan *node);<br />
<br />
<br />
<br />
== Functions of CustomExecMethods ==<br />
<br />
void (*BeginCustomScan) (CustomScanState *node,<br />
EState *estate,<br />
int eflags);<br />
TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);<br />
void (*EndCustomScan) (CustomScanState *node);<br />
void (*ReScanCustomScan) (CustomScanState *node);<br />
void (*MarkPosCustomScan) (CustomScanState *node);<br />
void (*RestrPosCustomScan) (CustomScanState *node);<br />
<br />
void (*ExplainCustomScan) (CustomScanState *node,<br />
List *ancestors,<br />
ExplainState *es);</div>Kaigaihttps://wiki.postgresql.org/index.php?title=Lookaside&diff=22283Lookaside2014-05-08T01:39:38Z<p>Kaigai: Created page with "= Purpose = The purpose of LOOKASIDE is to inform planner existence of alternative scan/join path on particular tables. Once planner know these alternative scan/join paths, it..."</p>
<hr />
<div>= Purpose =<br />
The purpose of LOOKASIDE is to inform planner existence of alternative scan/join path on particular tables.<br />
Once planner know these alternative scan/join paths, it can evaluate which is the cheapest path to run, in addition to the built-in logic.<br />
<br />
It is primarily considered to redirect to materialized-view transparently on reference to the source relations.<br />
But not all, Simon introduced several potential use cases.<br />
* KaiGai's work - https://wiki.postgresql.org/wiki/PGStrom<br />
* http://www.postgresql.org/message-id/52C59858.9090500@garret.ru<br />
* http://citusdata.github.io/cstore_fdw/<br />
* University of Manchester - exploring GPUs as part of the AXLE project<br />
* Barcelona SuperComputer Centre - exploring FPGAs, as part of the AXLE project<br />
* Some other authors have also cited gains using GPU technology in databases<br />
<br />
<br />
= Syntax =<br />
<br />
CREATE LOOKASIDE <name> ON <target relation><br />
TO <redirect relation>;<br />
<br />
CREATE LOOKASIDE <name> ON <target relation><br />
EXECUTE <path-generator function>;<br />
<br />
= FDW enhancement =<br />
<br />
* Replacement of Join/Aggregate by ForeignScan<br />
** Existing FDW mechanism does not support to replace Join node(s) by ForeignScan. Remote-join is a long-standing requirement for FDW, not only LOOKASIDE feature.<br />
** Also, Simon is planning to look at Aggregate replacement by ForeignScan.<br />
<br />
* ForeignScan without foreign-table<br />
** Existing ForeignScan assumes it performs in front of a particular foreign table; that means a cataloged relation is required. It potentially makes a trouble if we try to replace Join (or something other) nodes without cataloged relation by ForeignScan. So, KaiGai think we need to eliminate these restriction around ForeignScan.<br />
** Simon proposed to define foreign-table for each regular table, using event trigger. However, Stephen does not like this idea, and KaiGai worried about how does it works on Join.<br />
<br />
* MultiExec method support<br />
** Data exchange cost between exec-nodes is not ignorable. In case when two (or more) ForeignScan nodes are stacking, and their internal data format does not fit usual TupleTableSlot, it makes sense to exchange data as-is.<br />
** Existing MultiExec method in executor supports data exchange using custom data type on responsibility of parent and child node.<br />
*** Stephen commented, asynchronous data exchange is also important, not only bulk data-exchange.<br />
<br />
* ForeignScan with SubPlans<br />
** If an extension support its special logic to join relations, but don't want to support various method to scan relations, it is natural to leverage built-in scan logics (like SeqScan, ...).<br />
** KaiGai want ForeignScan to support to have SubPlans if FDW driver has capability.<br />
** Several core functions (that handles plan-tree recursively) needs to be exported to extensions, according to the experience in custom-plan works.<br />
<br />
= Requirement =<br />
<br />
Having an explicit linkage between data structures allows us to enhance an existing application by transaparently adding new structures, just as we already do with indexes. Specifically, that we allow more than one lookaside structure on any one table.<br />
<br />
* Explicit definition that we are attaching an alternate path onto a table (conceptually similar to adding an index)<br />
<br />
* Ability to check that the alternate path is viable (similar to the way we validate use of partial indexes prior to usage). Checks on columns(SELECT), rows(WHERE), aggregations(GROUP)<br />
** KaiGai agreed to its necessity, but doubt to support them in the first version.<br />
<br />
* Ability to consider access cost for both normal table and alternate path (like an index) - this allows the alternate path to *not* be chosen when we are performing some operation that is sub-optimal (for whatever reason).<br />
<br />
* There may be some need to define operator classes that are implemented via the alternate path<br />
<br />
* allows the join of one or more tables to be replaced with a single lookaside<br />
<br />
* LOOKASIDE on the writer commands (INSERT/UPDATE/DELETE).</div>Kaigaihttps://wiki.postgresql.org/index.php?title=9.4CF4Triage&diff=216859.4CF4Triage2014-01-10T23:42:07Z<p>Kaigai: /* Needs a Little Work */</p>
<hr />
<div>This page handles suggested patch triage for 9.4 final commitfest patches.<br />
<br />
==Good To Go==<br />
<br />
These patches are ready for commit as soon as the CF starts.<br />
<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1133 Add support to "IF NOT EXISTS" to others "CREATE" statements]<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1324 SSL: Show protocol and ciphersuite in server log]<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1325 SSL: libpq: Support TLSv1.1/TLSv1.2]<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1298 pg_stat_statements external query text storage]<br />
<br />
==Needs a Little Work==<br />
<br />
Smaller patches which can be included in 9.4 if<br />
they get a few hours of love from a committer or major hacker.<br />
<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1289 Hstore 2.0]<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1282 CustomScan API]<br />
<br />
==Big Patches==<br />
<br />
Very large, important patches which need a dedicated reviewer/committer for the duration of the commitfest. Ideally should list all dependant patches of each Big Patch.<br />
<br />
* Logical Changeset Extraction (link?)<br />
<br />
==Not Nearly Ready==<br />
<br />
Patches which need major work and/or spec discussions before commitment.<br />
<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1318 Store Extension Options]<br />
* [https://commitfest.postgresql.org/action/patch_view?id=1335 Standalone Synchronous Master]<br />
<br />
==WIP==<br />
<br />
Patches which are acknowledged not to be ready, or which are non-trivial patches which have not been introduced prior to CF4.</div>Kaigai