PgConf.Asia 2016 Developer Meeting

From PostgreSQL wiki
Jump to navigationJump to search

A meeting of the interested PostgreSQL developers is being planned for the morning of Thursday 1st December, 2016 in Tokyo, prior to PGConf.Asia 2016. In order to keep the numbers manageable, this meeting is by invitation only. Unfortunately it is quite possible that we've overlooked important individuals during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 9.6 release cycle. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies.

The afternoon will be a Developer Unconference, open to a wider audience.

This is a PostgreSQL Community event.

Meeting Goals

  • Review the progress of the 10.0 schedule, and formulate plans to address any issues
  • Address any proposed timing, policy, or procedure issues
  • Address any proposed Wicked problems

Time & Location

The event will be held on the fifth floor (using American/Japanese style counting) in room 5A at: 

 Akihabara Convention Hall
 Akihabara Dai Bldig 4F 1-18-13 Sotokanda, 
 Chiyoda-ku,
 Tokyo 101-0021, 
 Japan

Please see the website for details of how to reach the hall.

The morning session (9AM - 12PM) will be used for a structured meeting, and the afternoon session (1PM - 5PM) will be used for a 2 track mini unconference for invitees to the morning session and other interested developers.

PGConf.ASIA2016 Developer Unconference

RSVPs

The following people have RSVPed to the meeting (in alphabetical order, by surname) and will be attending:

  • Joe Conway
  • Etsuro Fujita
  • Magnus Hagander
  • Kyotaro Horiguchi
  • Kohei KaiGai
  • Bruce Momjian
  • Dave Page
  • Michael Paquier
  • Simon Riggs
  • Masahiko Sawada
  • Teodor Sigaev
  • Tomas Vondra
  • Amit Kapila
  • Takayuki Tsunakawa

Agenda

Time Item Presenter
09:00 - 09:10 Welcome and introductions Dave
9:10 - 9:20 10.0 Release Schedule (ref: https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Meeting#9:55_-_10:05_.09Next_Release_Schedule_.09All) All
9:20 - 9:35 Status report of the first two commit fests already done for PG10 development cycle. Michael
9:35 - 9:50 Reviewing unreviewed patches Simon
9:50 - 10:00 Revoking Committer access for inactive committers (ref: https://wiki.postgresql.org/wiki/PgCon_2015_Developer_Meeting#Committers) Simon
10:00 - 10:15 Providing information for applications which support PostgreSQL KaiGai/MauMau
10:15 - 10:30 Revisit varlena for larger data support (>1GB) KaiGai
10:30 - 10:45 Coffee break All
10:45 - 11:05 Transaction control for statement-level rollback and stored procedures MauMau
11:05 - 11:25 Multi-model database - beyond relational MauMau
11:25 - 11:45 The future of built-in Postgres sharding Bruce
11:45 - 11:55 Any other business Dave
11:55 - 12:00 Group photo All
12:00 Lunch

Agenda Items

Please list any agenda items below for inclusion on the schedule.

  • 10.0 Release Schedule (all) - see https://wiki.postgresql.org/wiki/PgCon_2016_Developer_Meeting#9:55_-_10:05_.09Next_Release_Schedule_.09All
  • Status report of the first two commit fests already done for PG10 development cycle. (Michael)
  • Providing information for applications which support PostgreSQL (KaiGai/MauMau)
  • The future of built-in Postgres sharding (Bruce)
  • Revisit varlena for larger data support (>1GB) (KaiGai)
  • Multi-model database - beyond relational (MauMau)
  • Transaction control for statement-level rollback and stored procedures (MauMau)
  • Reviewing unreviewed patches (Simon)
  • Revoking Committer access for inactive committers (not looking for a binding decision since not everybody is present) (Simon)

Minutes

Introductions:

Anti-clockwise, starting with Dave: Dave, Magnus, Joe, Masahiko, Michael, Amit, KaiGai, Takayuki (MauMau), Simon, Tomas, Kyotaro, Etsuro


10.0 Release Schedule:

Dave: [Review of timeline discussed in Ottawa]

Michael: We appear appear to be on track. Schedule looks to be in good shape:

Amit: We need to look at the individual big patches - are we on time with them?

Magnus: Do we want to push the release if we want some of the outstanding features?

Dave: If we want to push the release for a specific feature, there will be strong push back.

General opinion seems to be that we should not push the release - but if we want specific patches, we need to knuckle down.

Bruce: Are there any specific patches? Multivariate stats

Magnus: pg_logical

Bruce: Is there a sense that stuff is languishing in the commitfests?

Simon: MV stats is in that position.

Michael: MV stats has been pushed 3 or 4 times now.

All agree schedule looks good, there should (as usual) be no pushing of the release, and that we should select a new RMT at/following the FOSDEM developer meeting.


Status report of the first two commit fests already done for PG10 development cycle:

Michael: 1st commitfest was the largest ever, with 220 patches! Lots of new features submitted. At end of commitfest, noted that there were ~70 patches awaiting author. The 2nd commitfest finishes today. There are a lot of patches awaiting review, not so many waiting for author, and a couple waiting for committer. For the last couple of months, people have begun to use the CF app for tracking bug fix patches (which is good), but these are not receiving much attention. 12 patches are bug fixes ATM, only 5 are ready for committer.

There are a lot of small patches which people seem to like reviewing, whilst bigger patches get less attention. For example, the cmake patch, or WAL logging for hash indexes which is a 130K patch, or MV stats which is 500K(!)

Amit: Covering indexes is getting bounced from CF to CF.

Magnus: That one is getting returned with feedback though - it's going through the process.

Dave: Is it the norm that patches are getting bounced?

Amit/Michael: Yes, for the large complex ones.

Dave: So do commitfests still work?

Joe: The tracking is very important

Michael: Yes, the concept is very good. We can't stop people being uninterested in the reviewing, but it's great when it does work.

Bruce: Some of this is just really hard, grungy work - and people don't always want to do it.


Segway to: Reviewing unreviewed patches:

Michael: Before the segway, we should consider large patches that won't make it - I suggest that cmake is in that position.

Joe: We should have a way on the CF app to show if people actually want patches.

[side discussion on Cmake]

Magnus: One of the reasons for the FOSDEM meeting is that we can use that for final kicking out of patches.

Simon: This does follow on. I would welcome a list of things that we all want to go into the release. That's why I wrote PITR - it was at the top of the TODO list. [To Amit]: I'm happy to review your WAL/Hash indexes patch, but it hasn't changed since September and now has failures so I can't review it. We should have a CI system for ongoing testing of large patches. We should track large changes in GIT rather than using patches.

Magnus: Very few patches have git info on them in the CF app

Dave: We had this discussion at lunch yesterday - essentially, a superset of committers and trusted folks could mark patches as non-malicious, at which point they can be added to a branch for daily CI builds.

Simon: I just want to git-pull to get the latest version of your work, as some of these patches are huge. If we allow the patch author and committers to update a public branch, then others could submit patches to that - patch on patch. What I would like is nominate important/large patches and put them on a postgresql.org controlled git repo, which would allow them to be re-checked daily to avoid bit-rot. The author would be emailed if the patch fails, and would have access to that repo to correct bitrot. It is important that the first patch is submitted to the list, for all the normal reasons. After that we should be using the best modern tech to work collaboratively on important patches: git.

Dave: This is complex - we're not going to fix it here.

Joe: Let's discuss in the unconference.


Revoking Committer access for inactive committers:

Magnus: We have rules on this, passed at the 2015 meeting: "Dave proposed some rules on retiring inactive committers. 24 months zero commits, they get removed from the active lists. Passed."

Simon: I raise this here, as it's the first dev meeting in Japan and rules would affect a Japanese developer. I propose to implement the previously discussed rule which would affect Itagaki Takahiro - and be clear that it is a general rule and nothing personal.

Dave: Noted - this does need to be proposed through the private committers list.

  • ACTION ITEM: Simon to email priv-committers.


Providing information for applications which support PostgreSQL:

KaiGai: Please see the wiki page for "Ecosystem:Business Intelligence (BI)". This is useful info for people who want to choose a database system. I think we need a good central point of information about what systems work with PostgreSQL.

Dave: We have the Product Catalogue on the website. Why not use that?

KaiGai: I think it makes sense to maintain the central point of information, not just for applications, but also for use cases/references.

MauMau: Customers often ask if Postgres works with different software.

Dave: So if I understand correctly, the issues are application compatibility for which we have the product catalogue, and users/case studies for which we have a page on the website that is largely unmaintained.

Joe: It can be hard to get permission to release info like case studies from large companies.

KaiGai: Simon, do 2ndQuadrant have case studies?

Simon: My marketing people would be ecstatic if we put our case studies on postgresql.org

Tomas: We could just have a form that companies can update their info on

Dave: We have that, just without specific links for case studies.

Magnus: We have a political decision - who should be able to publish? What about Amazon RDS or Aurora? Where do you draw the line?

Simon: If it's on postgresql.org, we should copy the content to avoid abuse. Secondly - to be listed as a platinum sponsor, you must be able to provide 2 case studies (for example).


Revisit varlena for larger data support (>1GB):

KaiGai: Postpone to unconference, to get us back on time!


Transaction control for statement-level rollback and stored procedures:

MauMau: The motivation is to migrate users from other databases. We failed to aquire a particular customer following a benchmark - their legacy application used a legacy Cobol precompiler. We chose to use the Microfocus Cobol compiler. The application is essentially a batch application that issues many statements in a single transaction. The customer couldn't meet the performance requirements, one reason was that they needed statement level rollback. Both pgJDBC and ODBC emulate this using savepoints - but that has a high round-trip time. We want statement level rollback without using savepoints. What I want to ask here is, is it easy to implement statement level rollback?

Amit: What is statement level rollback?

Simon: He wants to use subtransactions around each statement. Use BeginInternalSubTransaction to do this, not Savepoints.

MauMau: Is it difficult to implement this in PostgreSQL?

Simon: I think it would be reasonable to have a parameter to allow the user to specify transaction behaviour.

Michael: Can you use Craig Ringer's libpq batch handling work?

Amit: It's better if you can propose a draft patch.

Simon: I think we can deal with this server-side with some changes in tcop. I'm writing some notes about it now for later discussion.

MauMau: Next is stored procedure support. Various people have tried to implement stored procedures

Dave: You're looking for different transactional control, rather than the calling convention which is the only different in the SQL standard from stored functions?

Joe: Right - the standard just differentiates on the calling convention, not the transactional side.

Simon: It doesn't really matter what they're called - the point is that users should have the ability to deal with transactions. It shouldn't be a major problem - we should be able to read the definition in one snapshot, then use more snapshots as needed whilst runnning the code.

Everyone seems to agree this is useful, but noone is working on this right now.

Joe: I don't think there's 100% agreement on what a stored proceudre in Postgres actually is.


Multi-model database - beyond relational:

MauMau: We are seeing more use of non-relational technologies these days. Is PostgreSQL going into other areas in order to become more popular? At the moment the DB rankings show Postgres at number 4, but I'm worried. Should we do more in document storage, KV, graph etc? Does the Postgres community want to enter into the wider database war?

Bruce: I think our community in general has done a poor job of adressing multi-modal, but we're so good at accepting ideas that we succeed. We've had people in the community who are visionary, and we;'ve been good at allowing them to expand. We've been successful, but we never really planned this.

Tomas: We never get a use case for a multi-modal database. We're not oppposed to it though.

KaiGai: People who have relational skills are more in demand. I think the key is the extendablility of PostgreSQL.

Magnus: I think the issue is mostly having a good API

MauMau: I heard in the past the PostgreSQL made a decision not to leave the relational world.

Bruce: I think you're thinking transactional control

Simon: Postgres is "beyond relational" so you're already there!

Tomas: The docs still call us an ORDBMS


The future of built-in Postgres sharding:

Bruce: Because of various people I've talked to, I got on the bandwagon a couple of years ago that we should include built in sharding. I'm very happy with the progress we've made. A lot of work has happened in Japan, and I wanted to ask if anyone has any comments or feedback.

Simon: I see a lot fo work happening, but I don't know why. If someone can explain why, then I might be able to back it. If I knew what you were doing, then I might help

Etsuro: I don't have an idea of the overall picture of sharding, but my understanding is basically based on FDWs. For now, we are focussing on building such building blocks, and after improving such features, we can discuss the future of the overall architecture.

Simon: If we have something to talk about now, I'd like to talk about it for 10.0. I'd like to discuss this this afternoon.

Bruce: NTT maintained XC for 10 years for little gain - most of that problem was the forked code. NOW, we should be able to get the benefits with sharding.

Joe: Is there a wiki page for this, listing all the building blocks and their status?

Bruce: I have a file on my laptop, as I've experienced so much negativity.

Simon: We need a clear statement on what people are trying to do, even if they're trying to do it in different ways. I want a detailed discussion on what we're trying to do.

Dave: Is there anything on the wiki at all Bruce?

Bruce: No

Dave: How about creating a wiki page with your vision on it, listing the goals and how the building blocks will fit together to meet those goals. Put a disclaimer on it so people don't think it's official or an EDB roadmap.

Tomas: I can't discuss it with you sensibly without a proposal. I'm concerned that we'll damage foreign data wrappers in a bid to make them more suitable for sharding.

Retrieved from "https://wiki.postgresql.org/index.php?title=PgConf.Asia_2016_Developer_Meeting&oldid=37560"