PgCon2014ClusterSummit

Clustering and Replication Developers Summit pgCon 2014

Tuesday, May 20th

2PM to 6PM

Preceded by PostgresXC Meeting 9AM to 1PM

Followed by the PostgresXC Pizza Demo at 7pm.

University of Ottawa

Room: University Center Room 205

Sponsored by NTT Open Source

Agenda

1:00PM to 2:00PM

Lunch supplied for anyone coming for the whole day (PostgresXC + Clustering)

Due to a sudden increase in numbers, only the 15 participants staying for the full day (both XC and Clustering) will be given lunch. People attending the clustering summit only should eat their own lunch, and arrive at 2pm.

1:30PM to 2:00PM

Please bring any last-minute agenda items to Josh Berkus at this time.

2:00PM to 2:45PM

Introductions, and status reports from Replication/Clustering Projects:

Status updates (please volunteer below if you can give an update):

• pgPool II: Tatsuo Ishii
• Postgres-XC: Koichi Suzuki
• Built-in Replication: ????
• Postgres-XL: Mason Sharp
• Stado: Mason?
• Bucardo: ???
• LSR & BDR: ???
• Slony: Chris Browne/Jan Wieck

If you are at the summit representing a specific replication or clustering tool, you should prepare a 1-3 minute summary of current progress and issues. If you want to use slides, please provide slides in PDF form to Josh Berkus by Friday, May 16.

2:45PM to 3:15PM

Summary of Clustering API projects.

Summit attendees who have been working on core clustering features should give an update as to progress and current issues. Please present a 5-10 minute summary. Attendees may use their own laptops for slides, or give slides to Josh Berkus.

Topics:

• Logical Replication (not present)
• Exportable Parser/Planner: no activity for year. Some discussion.
• FDW and SQL/MED
  • Function scan push-down
  • Node partitioning with constraint exclusion

3:15PM to 3:45PM

Break

3:45PM to 4:30PM

Summary of Clustering API projects, continued.

4:30PM to 6:00PM

Discussion of priorities, progress and ideas for core clustering projects and APIs.

Goal of this discussion is to modify the list of core clustering features and get commitments for hackers to work on specific features. Also, to supply discussion items for the following day's Developer Meeting

RSVP List

1. Josh Berkus
2. Koichi Suzuki
3. Tatsuo Ishii (SRA OSS)
4. Haruka Takatsuka (SRA OSS)
5. Yugo Nagata (SRA OSS)
6. Mason Sharp
7. Bruce Momjian
8. Chris Browne
9. David Wheeler
10. Moshe Jacobson (Nead Werx)
11. Chris Autry (Nead Werx)
12. Joe Conway
13. Steve Singer
14. Jan Wieck
15. Jim Mlodgenski
16. Vik Fearing
17. Marc Linster (EDB)
18. Ahsan Hadi (EDB)
19. Zahid Iqbal (EDB)
20. Etsuro Fujita (NTT OSS Center)
21. Amit Kapila
22. Krzysztof Nienartowicz (ESA)
23. Hitoshi Hemmi (NTT OSS Center)
24. Álvaro Hernández
25. Motoyuki Kawaba
26. Haribabu Kommi
27. Yurie Enomoto

Meeting Notes

Introductions

We did not capture names of attendees. Simon Riggs and Andres Freund were not able to make the meeting.

Projects

• pgPool II: Yugo Nagata
  • 3.4 improvements, adds PG exception and memory manager
  • slides

• pgPool II: Ahsan Hadi
  • performance testing
  • scaling improvements for 1-5 replicas and 75% read workload

• Postgres-XC: Koichi Suzuki
  • release 1.2 items
  • more openness for volunteers, committers, roadmap

• Stado: Mason Sharp
  • nothing big to report
  • move from Launchpad to Sourceforge

• Postgres-XL: Mason Sharp
  • optimizations over Postgres XC
  • focus on stability, performance, and openness

• Slony: Chris Browne/Jan Wieck
  • Slony top added
  • better tooling
  • add logical replication support to avoid trigger overhead, have prototype

• LSR & BDR: Mark Sloan
  • Intel funding
  • still work for 9.5, e.g. DDL

Clustering API Projects

• Passing Query parse trees to other nodes
  • no one volunteered to work on it

NTT is working on some features for FDWs:

• FDW
  • add syntax to allow FDW partitions/inheritance
    • this was intended for 9.4, but got pushed back
  • support constraint_exclusion across shards
• FDW pushdowns
  • JOINS
  • Aggregates
  • Scans
  • Contributing XC/XL code to mainstream PostgreSQL.

New Projects

The meeting discussed a number of possible new projects. The below lack anyone qualified to work on them:

• Async FDW execution
• Parallel Query?
• GTM
  • Hooks for the GTM?
  • Two-node sync?
  • WAL barrier for shards
    • Problem with transaction commit order visibility differences between master/slave
• Partition primary keys/foreign keys
  • either using the partition index, or not
  • other multi-table constraints

There was some discussion of the new BDR features, hampered by the absence of the BDR team. This included:

• Adding more simple BDR formats to core/contrib
• Adding a core/contrib LSR apply plugin, maybe based on Slony copy.

We discused the need for multiple host connections for HA configurations. For example, if you have a pgpool and a failover pgpool, you'd like to be able to connect to both. Jim Mlodjenski committed to working on a multihost connection change for libpq, which would look like:

   database=postgres host=10.0.0.1,10.0.0.2,10.0.0.3 port=5432

PostgresXC Pizza Demo

Koichi Suzuki demonstrated setting up and managing PostgresXC with the new pgxc_ctl. This program, despite its name, is a full command shell for a PostgresXC cluster, supporting node changes, sharding commands, and status checks.