Pgcon2014PostgresXCmeeting

PostgresXC Developer Meeting pgCon 2014

Tuesday, May 20th

9AM to 1PM

Followed by Clustering Summit at 2PM

University of Ottawa

Room: University Center Room 205

Sponsored by NTT Open Source

PostgresXC Day

9AM to 9:30AM

Introductions and Coffee

Introduction: Koichi Suzuki

9:30AM to 11:00AM

Presentations:

• PostgresXC: current status recent events
• PostgresXC at the European Space Agency (Krzysztof Nienartowicz)
• Postgres-XL Overview (Mason Sharp)

11:00AM to 1:00 PM

PostgresXC Development and release discussion.

1PM to 2PM

Lunch for those attending both the PostgresXC summit and the Clustering Summit

Due to late RSVPs, this will be limited to those attending the full day ONLY.

6:30PM to 8:30PM

PostgresXC Demo + Pizza

Held in the same room.

RSVP List

RSVP if you plan to attend. Please RSVP for this and the Clustering Summit if you plan to attend both, and want lunch.

1. Koichi Suzuki
2. Josh Berkus
3. Mason Sharp
4. Bruce Momjian
5. Rafael Guembes
6. Jim Mlodgenski
7. Mark Linster
8. Zahid Iqbal
9. Ahsan Hadi
10. Hitoshi Hemmi
11. Motoyuki Kawaba
12. Haribabu Kommi
13. Yurie Enomoto

PostgresXC Meeting Notes

Attendees

• Koichi Suzuki, leading PostgresXC for several years.
• Galy Lee, Huawei/Postgres China
• David, also from Huawei/Postgres China
• Hitoshi Hemmi, NTT, PostgresXC project
• Hari, Fujitsu Australia
• Kawaba, Fujitsu Laboratories
• Yurie, Fujitsu Australia, Database engineer
• Mason Sharp, Translattice, PostgresXC project
• Bruce Momjian, pg Core Team, EDB
• Jim M., OpenSCG
• Ahsan Hadi, EDB, PostgresXC project
• Zahdi, EDB, xDB Replication Server
• Vik Fearing, Dalibo
• Marc Linster, EDB
• Moshe Jacobsen
• Chris Autry
• Rafael Gambez
• Jim Nasby, Enova Financial
• Krzysztof Nienartowicz, European Space Agency

Presentations

First, Koichi welcomed everyone to then 2nd meeting at pgCon.

Krzysztof Nienartowicz did a presentation on ESA Gaia Project at the European Space Agency. Gaia is a project to map all objects in the night sky over a period of years at much higher resolutions than was ever possible before. This uses a special new mirror-based satellite telescope developed for Gaia, and generates terabytes of data per day. Since they are already analyzing the data with PostgreSQL, they hope to use PostgresXC in order to build up a data platform to hold their huge amonts of sky object data to analyze it. He discussed some of the requirements they have for PostgresXC to start using it later this year. Slides here:

Mason Sharp talked about Postgres-XL, a fork of PostgresXC. Slides Here:

According to Mason, they are focusing more on analytical side, not OLTP. They are currently on 9.2.

• Analytical vs. OLTP
  • XL has disabled triggers
  • Some limitations on cursors
  • Planner & Executor is different ... pushes more work down to nodes, doesn't use controller.

XL code is marked out by additional IFDEFS. There is also stuff being merged into Translattice.

StormDB was a proprietary fork of PostgresXC. Translattice released it as separate project under the MPL. Bruce raised the issue that XL will divide efforts when there already too few developers; it doesn't seem to make sense to create separate communities. Main difference is focus on planner/optimizer. XL project is more focused on stability and bug-fixes than XC project; puts fixes ahead of features, XC project does the opposite.

So one of the big issues is a disagreement in the XC community over stability vs. features. Discussion ensued.

XC Status and Development

Koichi then presented on the current status of PGXC. Slides available here:

• most of pgxc consists of a set of c files in addition to postgres files
• many changes in mainline Postgres code, bracketed by #IFDEFs.

1.0 was based on Postgres 9.1.

Koichi called for more feedback from potential users.

Community Questions

Bruce brought up the question of when will PostgresXC be production-ready. This means that they've been focusing on features without worrying about releases. Bruce things we should push for a production-ready solution now instead of focusing on feature development.

Need to share issues among prospect users and testing users. Internal NTT users don't like to share issues around confidential databases. There was some discussion of community involvement issues. Bruce suggested that PostgresXC needs to be a "real" OSS project now. Jim Nasby argued that PostgresXC needs to make stability a priority always, not a roadmap item.

Ahsan suggested that we need to have a bug tracker for PostgresXC which is public.

Jim and Ahsan also discussed that the pushdown code from XC should be contributed to FDWs in mainstream Postgres.

Bruce asked if outside developers can become committers on the project. This still needs discussion.

Future Development

There was discussion of the role of the Global Transaction Manager. However, overhead for getting synchronized snapshots across nodes is very high. Heikki is working on a commit ordering improvement for 9.5 which may help this.

There was discussion about global constraints and partitioning over multiple servers. Koichi said that PostgresXC assumes a sort of star schema, with partitioned fact table and duplicated dimension tables.

1.3 Roadmap and Beyond

• Merge PostgreSQL 9.4
• Integrate coordinator and datanode so that they don't have 3 different binaries.
• Global Constraints (maybe not per discussion)
• GTM/GTM proxy integration into Postgres backend
• Subquery optimization
• Advanced cursor features.
• Fixes for pgxc_ctl

Mason questioned whether the coordinator/datanode integration would be a lot of effort and not worth it. Josh questioned if the advanced cursor features were worth it.

Ahsan suggested that stability should be the focus of 1.3 for PostgresXC.

Galy Lee discussed some problems they have been having with XC. Koichi discussed statement vs. plan push-down. Transferring the statement is actually much smaller than transferring the plan.

Josh questioned the wisdom of going after DBT5, because it's specifically designed to be lock-heavy and to perform poorly on multinode databases. Also, the DBT5 benchmark is buggy.

NTT will publish the suggested priorities for 1.3 for the online community to comment on.

PostgresXC has 470,000 lines of code different from mainstream Postgres. Most of that is the planner.

Summary

In general, we have decided that the current feature set of PostgresXC is adequate, but we should be working much more on stability and performance. And we should make the community work much more open so that more people can be involved. We should also export more PostgresXC features so mainstream Postgres.