PgCon 2017 Developer Meeting

A meeting of the interested PostgreSQL developers is being planned for Tuesday 23 May, 2017 at the University of Ottawa, prior to pgCon 2017. In order to keep the numbers manageable, this meeting is by invitation only. Unfortunately it is quite possible that we've overlooked important individuals during the planning of the event - if you feel you fall into this category and would like to attend, please contact Dave Page (dpage@pgadmin.org).

Please note that the attendee numbers have been kept low in order to keep the meeting more productive. Invitations have been sent only to developers that have been highly active on the database server over the 10.0/9.6 release cycles. We have not invited any contributors based on their contributions to related projects, or seniority in regional user groups or sponsoring companies.

As at last years event, an Unconference will be held on Wednesday for in-depth discussion of technical topics.

This is a PostgreSQL Community event.

Meeting Goals

Define the schedule for the 11.0 release cycle
Address any proposed timing, policy, or procedure issues
Address any proposed Wicked problems

Time & Location

The meeting will be:

9:00AM to 12PM
DMS 3105
University of Ottawa.

Coffee, tea and snacks will be served starting at 8:45am. Lunch will be after the meeting.

RSVPs

The following people have RSVPed to the meeting (in alphabetical order, by surname):

Aleksander Alekseev
Oleg Bartunov
Joe Conway
Jeff Davis
Andrew Dunstan
Peter Eisentraut
Andres Freund
Stephen Frost
Etsuro Fujita
Peter Geoghegan
Kevin Grittner
Robert Haas
Magnus Hagander
Álvaro Herrera
Kyotaro Horiguchi
Amit Kapila
Haribabu Kommi
Alexander Korotkov
Tom Lane
Amit Langote
Heikki Linnakangas
Noah Misch
Bruce Momjian
Thomas Munro
Dave Page
Michael Paquier
Masahiko Sawada
Teodor Sigaev

Apologies

Simon Riggs

Agenda Items

Upgrading PostgreSQL without a downtime. (Aleksander Alekseev)
Commit fest management (Michael)
Intellectual Property Issues (Andres)
Recognizing Contributors (Stephen)
Release notes scope, and giving credit (Peter E.)
Web site design (Peter E.)
64-bit xids (Alexander Korotkov)
Adopting an indent fork (Álvaro Herrera)
Please add suggestions for agenda items here. (with your name)

Agenda

Time	Item	Presenter
09:00 - 09:10	Welcome and introductions	Dave Page
09:10 - 09:20	PG 11 release and commitfest schedule	Dave Page
09:20 - 09:40	Commit fest management	Michael
09:40 - 10:00	Upgrading PostgreSQL without a downtime	Aleksander Alekseev
10:00 - 10:05	Intellectual property issues	Andres
10:15 - 10:30	Web site design	Peter Eisentraut
10:30 - 10:45	Coffee break	All
10:45 - 11:00	Release notes scope, and giving credit	Peter Eisentraut
11:00 - 11:20	Recognising contributors	Stephen
11:20 - 11:40	64-bit xids	Alexander Korotkov
11:40 - 11:50	Adopting an indent fork	Álvaro Herrera
11:50 - 12:00	Any other business	Dave Page
12:00	Lunch

Minutes

Welcome and introductions

Attendees:

Aleksander Alekseev
Joe Conway
Jeff Davis
Andrew Dunstan
Peter Eisentraut
Andres Freund
Stephen Frost
Etsuro Fujita
Peter Geoghegan
Kevin Grittner
Robert Haas
Magnus Hagander
Álvaro Herrera
Kyotaro Horiguchi
Amit Kapila
Haribabu Kommi
Alexander Korotkov
Tom Lane
Amit Langote
Heikki Linnakangas
Noah Misch
Bruce Momjian
Thomas Munro
Dave Page
Michael Paquier
Masahiko Sawada
Teodor Sigaev

PG 11 release and commitfest schedule

Haas proposed using the same schedule as last year, only shifting the dates forward by one year. Measure passed unanimously.

Timing of the release branch is not yet clear.

Commit fest management

(Paquier) I seek feedback on recent CF management. At the end of each CF, I classify each patch's status and send a short email to the patch's thread. Is that spam? (Haas) No, it's good; some authors read their threads only. Don't decide the fate of a patch on an additional thread that doesn't include the author. CF manager does need to push other CF participants.

(Freund) Don't always move patches to the next CF; use Returned with Feedback instead. Patches have traversed 4+ CFs without changing. (Haas) If something is or should be Waiting on Author at CF end, use Returned with Feedback. When authors have submitted an updated patch at the end of a CF, Paquier has moved it to the next CF. That is a good move. (Lane) I worry about patch authors not familiar with the CF process, who may not know to resubmit after Returned with Feedback. I propose moving the patch at the end of one CF, then closing it if the next CF starts with no state change.

(Freund) Returned with Feedback breaks up the history, because the next submission will create a new entry. (Eisentraut) That is a tooling problem.

(Paquier) CFs used to have 100 patches max, but they now see 250. Process struggles to keep up. (Haas) If we ever kept up with patch flow, we don't now.

(Grittner) Patch repeatedly moving from one CF to the next can mean nobody was interested, or nobody competent had the time. (Freund) We're bad at giving early feedback that a change is not wanted. (Lane) For every patch, at least one person (the author) found it interesting. (Geoghegan, Lane) We don't outright reject very much. (Kapila) Each case is controversial. (Misch) Most almost-rejections are due to implementation, which we can't foresee early.

(Eisentraut) When we move a patch to the next CF, the reviewer may not be aware. Should we clear the reviewer slot on move? (Haas) Reviewer list in CF app is often very wrong. (Paquier) Eliminate CF app tracking of reviewer? (Kapila) Reviewer field is helpful as a target for CF manager nagging. (Haas) Don't remove it, but don't trust it. (Lane) Should the CF manager aggressively remove an inactive reviewer? (Haas) I support clearing the field at CF end. (Grittner) When removing a reviewer, notify that person by email. (Linnakangas) The open items process, with Misch emails, has worked perfectly. (Eisentraut) I don't care about nag emails. Patches get stalled ~6 weeks because the listed reviewer is idle; other reviewers focus on entries with blank reviewer.

(Paquier) The wiki's CF checklist is very outdated. I propose to rewrite it. Will add useful CF app links to that page or to the patch submission page. After doing those updates, I will request feedback from -hackers.

Upgrading PostgreSQL without a downtime

(Haas) As a technical topic, this is out of scope for this meeting. Save it for the unconference. (Page) Okay to discuss technical decisions such as openness to on-disk format changes. Technical discussion did work well in Brussels.

(Korotkov, Alekseev) The community is conservative. For example, it uses Perl. We plan to allow v10->v11 upgrades but not direct v10->v12 upgrades. Is that okay? (Misch) I don't see a problem with this one-version upgrade window. (Hagander) Agreed; it's okay to need five upgrades to cross five versions.

(Haas) Biggest thing that won't work is system catalog changes. (Geoghegan) Logical replication solves many of the problems.

(Page) With SQL Server, you can just upgrade the binaries and restart.

(Alekseev) I'm defining "zero downtime" as "indistinguishable from failover to a standby". (Grittner) Not proposing to preserve TCP connections? (Alekseev) Correct, not proposing that. (Geoghegan) Suggest starting with minor release updates. (Haas, Grittner) Under this definition of zero downtime, that's already possible.

(Haas) I posted a pie-in-the-sky design for major releases. Lane found flaws.

(Linnakangas) This will differ from the multiple-page-versions approaches, which still assumed use of pg_upgrade.

(Page) Is it acceptable to require replication and a second server?

(Herrera) Will every catalog change require code to support this? (Alekseev) No. For example, we just won't drop old catalog columns. (Davis) Challenging thanks to our use of C structures and offsets to access catalogs.

(Haas) It will never work to perform physical replications across major versions. Could have v11 binaries recognize and convert a v10 cluster, but cannot have a v10 replica with a v11 master. (Freund, Momjian) Yes; the protocol is compatible, but the payload is not. (Linnakangas) It's possible but far more complicated and hard to test.

(Misch) Propose conclusion that the group has no objections to this proceeding. We've provided various challenges to cover in the design.

(Lane) Possible alternative is to spend time making pg_upgrade faster. (Freund) ANALYZE statistics are the hard part. (Haas, Eisentraut) pg_upgrade fails for hard-to-predict reasons.

Intellectual property issues

The group voted to exclude this topic from the minutes.

Web site design

(Eisentraut) Site is stuck in the past. We have occasional pushes to hire someone or otherwise fix it. (Frost) We did a bunch of work that didn't get merged. Showed it off at pgconf.eu 1.5 years ago. Sarah Conway of Crunchy Data is leading the design effort. Someone needs to reimplement that on a different Django version. (Momjian, Haas) This lacks project management. Who, if anyone, will do the remaining work? (Frost) I am targeting completion this fall. (Eisentraut) If I email pgsql-www every two weeks, is that annoying enough?

(Misch) What's actually wrong with the current site? (Eisentraut) Looks bad on small devices. (Geoghegan) It creates a bad impression for new users.

(Page) Incremental changes can be hard; changing the font size could lead to months of bikeshedding. (Hagander) Partial migration is undesirable, because it would leave the site with a blend of different designs.

(Hagander) Updating the wiki is more urgent. (Eisentraut) Is there a backlog of infrastructure issues? (Hagander) Just the wiki and the mailing lists.

Release notes scope, and giving credit

(Eisentraut) Some items attributed to me warrant credit to important reviewers.

(Linnakangas, Eisentraut) How should we treat performance improvements in the release notes? (Momjian) I can easily recognize significant feature additions, but it's harder to tell the significance of a performance improvement. I want to include changes that are actionable, that allow new uses of PostgreSQL. From 1200 commits I wrote 200 release note entries, some of those associated with more than one commit. Adding commit hashes as SGML comments has been a big win. I expect changes to initial notes; for v10, the editing thread was shorter than usual. (Freund) I was dissatisfied with the original treatment of performance patches and the general statement about their value. "Will it run faster?" is the #1 thing for my users. However, I'm content with the release notes as they stand today. (Haas) Committers have an action item to write commit messages that explain the change's significance. (Momjian) Commit messages were really good. (Haas) Hard to write notes for Lane improvements that address rare query shapes. (Linnakangas) Should consciously include an end-user-visible example.

(Grittner) Most releases improve two kinds of performance. First, they improve concurrency: they move the performance "knee" to the right (more clients) and make its decline shallower. Second, they make certain workloads suffer less bloat.

(Conway, Haas) Including mailing list links in commit messages has been useful.

(Haas) Do we keep including author names in release notes? (Freund) Linking to the commit would suffice for credit. (Eisentraut) Should we move all names to the end? (Momjian) Names at the end is okay if we do link to commits. (Haas) Linking to commits would be useful. (Momjian) Use javascript to show/hide all commit references, and/or show them on mouseover. (Dunstan) PDF docs won't handle that as well. (Eisentraut) Implementing dynamic behavior in every doc format is a lot of work.

(Momjian) v10 has 10-20% fewer release note items than recent releases, but it has big items. (Andres) That will change next release.

(Momjian) For 9.2 and 9.3, Josh Berkus and others gave feedback that listing reviewers was too much. (Eisentraut) Nonetheless, we should list them in the release notes. (Misch) I realize it's not the community consensus, but I remain supportive of listing reviewers at the bottom of the release notes. (Dunstan, Haas) We wouldn't mind that. (Dunstan, Misch) We don't credit one-line reviewers in our commits, so they'd be appropriately excluded from release notes. (Linnakangas) We could even credit everyone who send a mailing list post. (Geoghegan) I favor inclusion: not one-line reviewers, but almost anything else. People don't read notes end-to-end, anyway. (Eisentraut) Propose we collect all names mentioned in commit messages and stick them at the end of the v10 release notes. I will do the work. Measure passed unanimously.

Recognising contributors

(Frost) We have competing proposals for updating the web site contributor list. Need to define criteria. Could even just use the list Eisentraut makes for the release notes. (Haas) Don't want to add that many. When I proposed a list, I had forgotten that last year's meeting had commissioned another effort. Committed CF entries dominated the Frost proposal, and that's a problem. (Frost) I looked at email size as a way to recognize non-code contributions. (Haas) We do want to recognize non-code contributions.

(Freund) We're letting perfect be the enemy of good. (Page) We should vote yea/nay on a list presented so far, and move on. (Haas) Analyzing 2016 commits took ~8hr. Fine to automate if we can, but automation is inessential. I'm willing to spend 24hr analyzing satellite projects. Frost has blocked each of my proposals. (Page) Your time is better spent on code. (Haas) I thought that for five years, too. (Page) Propose taking Conway's list, discussing anything needed, and adopting it.

(Page, Haas, Geoghegan) Numbers should be a starting point; they should not dictate the final answer. (Haas) Best to remove CF data from the pile.

Poll: ~40% of this room is not on the contributors list.

(Geoghegan) We'd be unanimous on recognizing 10-20 people.

(Linnakangas) Who will physically update the site? (Conway, Lane) Suggest converting Haas list to a web site diff and sending it to pgsql-private-committers@ for approval. (Haas) Expect more cycles of update. For example, I only classified people as past contributors when 100% confident. Complaints are inevitable.

(Haas) Propose having Frost and myself sit down after this meeting and agree on something. (Momjian) I want the site updated by Thursday. (group conclusion) Haas and Frost may proceed to update the list, letting let this happen quickly.

64-bit xids

(Korotkov) Postgres Pro has a patch for this. Some customers have problems not fixed by 9.6 freeze work. No access to production systems for seeing details.

(Misch) Community would need to see the good-case and worst-case benchmarks.

(Grittner) This won't fix vacuuming to reclaim space.

(Lane) Eliminating freezes is attractive.

(Geoghegan) What happened to the Linnakangas version of this? (Linnakangas) It's unfinished.

(Haas) We know there's more to improve here, but we don't know what exactly remains.

(Haas) I will vote against widening xmin/xmax on disk. The Linnakangas approach didn't have that problem. (Linnakangas) Don't want on-disk format changes.

(Linnakangas) Do problems remain despite free space map and freeze map? (Freund, Geoghegan) Yes. (Freund) I see anti-wraparound vacuum problems in 9.6 three times per week, in response to configuration changes. Spending 1.8B XIDs in two days is not the worst I've seen. (Geoghegan, Freund) Nobody cares about clog size. (Haas) People do care, but not the same people who care about anti-wraparound vacuum.

Adopting an indent fork

(Lane) Considering adoption of Piotr Stefaniak indent fork. I studied the diff it causes and liked nearly everything. Should we embed it in the PostgreSQL tree, or should we give it a standalone repository? In-tree lets us correlate changes, and out-of-tree lets us use the latest version on all branches. (Haas) Weird to cram a standalone tool into the main tree. (Misch) Similar to TAP support modules. (Lane) Similar to much of src/tools.

(Freund) Great if every contributor can easily run indent. (Eisentraut) Developers will figure it out easily, relative to other challenges of starting development like installing dependencies and configuring one's editor. (Freund) Mainstream OSs package all dependencies, so that part is easy.

(Haas) We should reindent every week or every day. By end of cycle, the tree has accrued lots of divergence. It's hard to commit indent-clean code to a file that is already not indent-clean. "make indent" would be great.

(Eisentraut) If we switch indent implementations, I want one with upstream maintenance. (Freund) Author of proposed replacement is a FreeBSD committer and plans to submit indent code to FreeBSD. (Misch) We won't pull from upstream automatically; syncs will be more like timezone code syncs. (Freund) The replacement is already more-maintained than pgindent. (Lane) I vote not to wait for better maintenance. (Haas) If Lane will cover any problems, let it happen. Proposed tool does improve things.

(Eisentraut) Happy leaving v10 as-is (pgindent).

(Lane) Distinct issues (1) whether to switch indent tools and (2) how to distribute indent tool code. (Misch) Also (3) how often to reindent.

Any other business

(Misch) Congested agenda today. Should we meet longer next year? (Haas) Schedule was based on an unconference starting after the meeting, which we no longer do. But we wouldn't survive a 09:00-17:00 meeting. (Hagander) Perhaps 09:00-14:00?

(Page) How should the other developer meetings compare to this one? Brussels has been heavy on technical topics. (Haas) Unconference is better for technical topics. We used to have lots of narrow-interest technical topics that boiled down to the presenter speaking at a non-participating group.

(Linnakangas) When linking to mailing list posts in commit messages, I use a postgresql.org URL, but most use postgr.es. (Eisentraut) Should we link to the whole thread or to a single message?