PostgreSQL wiki - User contributions [en]

CustomCumulativeStats

2026-05-06T09:04:31Z

Dpage:

= Custom Cumulative Statistics (pgstats) =

Extensions can register their own custom cumulative statistics to the gathering of any statistical data using the same APIs as PostgreSQL's [https://www.postgresql.org/docs/devel/monitoring-stats.html backend pgstats].

Each custom cumulative statistics needs to register using a unique ID (across all extensions) to prevent problems with conflicts when handling the stats data (hash key in dshash table, read and write of pgstats file). The IDs are limited at compile-time, however, and only IDs between PGSTAT_KIND_CUSTOM_MIN and PGSTAT_KIND_CUSTOM_MAX (inclusive) are available natively for custom cumulative statistics as of PostgreSQL 18, as defined in utils/pgstat_kind.h. It is possible to change these limits.

This page lists the current extensions that require custom cumulative stats, and the IDs which are reserved for them. If your extension requires a new type of custom cumulative statistics, start out by using PGSTAT_KIND_EXPERIMENTAL (24 as of PostgreSQL 18). When the project is close to release or being used in production, choose an unused stats kind ID and add to this list to reserve it.

End-users must ensure that the extensions they deploy do not conflict, while it's not expected with open source extensions for which authors must take care of the current list, close source extensions could easily overlap.

= List of Custom Stats =
{| border="1" cellpadding="3"
!KindId||Extension||Description||License
|-
|24||N/A||Used for extensions still in development||N/A
|-
|25||N/A||In-core test module for variable-numbered stats||N/A
|-
|26||N/A||In-core test module for fixed-numbered stats||N/A
|-
|27||[https://github.com/pganalyze/pg_stat_plans pg_stat_plans]||Aggregated plan statistics tracking||PostgreSQL license
|-
|28||[https://github.com/fabriziomello/pg_stat_log pg_stat_log]||Log messages tracking||PostgreSQL license
|-
|29||[https://github.com/pgEdge/spock/ spock]||Replication stats tracking||PostgreSQL license
|}

CustomWALResourceManagers

2025-09-23T08:41:12Z

Dpage: Add reservation for Spock.

= Custom WAL Resource Manager (rmgr) =

Extensions can register their own custom WAL resource managers to support custom redo, physical replication, and logical replication/decoding.

Each resource manager needs to register using a unique ID (across all extensions) to prevent problems with crash recovery and replication. The IDs are limited, however, and only IDs between 128 and 255 (inclusive) are available for custom resource managers.

This page lists the current extensions that require a custom resource manager, and the IDs which are reserved for them. If your extension requires a new custom resource manager, start out by using RM_EXPERIMENTAL_ID (128). When the project is close to release or being used in production, choose an unused resource manager ID and add to this list to reserve it.

End-users must ensure that the extensions they deploy do not conflict, while it's not expected with open source extensions for which authors must take care of the current list, close source extensions could easily overlap.

= List of Custom Rmgrs =

{| border="1" cellpadding="3"
!RmgrId||Extension||Description||License
|-
|128||N/A||Used for extensions still in development||N/A
|-
|129||[https://github.com/orioledb/orioledb OrioleDB]||OrioleDB – building a modern cloud-native storage engine (... and solving some PostgreSQL wicked problems) 🇺🇦 ||[https://opensource.org/license/postgresql/ PostgreSQL License]
|-
|130||[https://enterprisedb.com EnterpriseDB]||Bluefin – Compressed IoT storage|| Proprietary
|-
|131||[https://enterprisedb.com EnterpriseDB]||Bluefin2 – Compressed IoT storage|| Proprietary
|-
|132||[https://enterprisedb.com EnterpriseDB]||Marlin – High Update storage|| Proprietary
|-
|133||[https://enterprisedb.com EnterpriseDB]||Marlin2 – High Update storage|| Proprietary
|-
|134||[https://github.com/neondatabase/neon/ Neon]||Neon storage system|| [https://www.apache.org/licenses/LICENSE-2.0.html Apache 2]
|-
|135||[https://postgrespro.com PostgresPro]||AQO – Adaptive query optimization|| Proprietary
|-
|136||[https://postgrespro.com PostgresPro]||sr_plan/pgpro_multiplan – Save and restore plan|| Proprietary
|-
|137||[https://github.com/paradedb/paradedb/tree/dev/pg_analytics pg_analytics]||pg_analytics extension|| [https://www.gnu.org/licenses/agpl-3.0 GNU AGPL3]
|-
|138||[https://github.com/pgroonga/pgroonga PGroonga]||PGroonga - Fast full text search index for all languages|| [https://opensource.org/license/postgresql/ PostgreSQL License]
|-
|139||[https://enterprisedb.com EnterpriseDB]||Hsphere - Hypersphere AI embedding index|| Proprietary
|-
|140||[https://github.com/percona/pg_tde Percona]||pg_tde - data at rest encryption extension|| [https://opensource.org/license/mit/ MIT]
|-
|141||[https://www.thenile.dev Nile]||Nile Storage|| [https://www.apache.org/licenses/LICENSE-2.0.html Apache 2]
|-
|142||[https://postgrespro.com PostgresPro]||pgpro_usage – per-user statistics on accessing relations and functions|| Proprietary
|-
|143||[https://postgrespro.com PostgresPro]||pgpro_result_cache - an extension for query result caching|| Proprietary
|-
|144||[https://github.com/pgedge/spock pgEdge]||spock - Logical multi-master PostgreSQL replication|| [https://opensource.org/license/postgresql/ PostgreSQL License]
|-
|241||[https://github.com/citusdata/citus/tree/master/src/backend/columnar Columnar]||Columnar compression extension||[https://www.gnu.org/licenses/agpl-3.0 GNU AGPL3]
|}

[[Category:Extensions]]

Committers

2024-03-01T09:56:58Z

Dpage: /* New Committers and Removing Committers */

This is the current list of people with access to push to the git repository with their user names. For technical details on how committing works, see [[Committing with Git]]. Note: This is just a list of people who currently have access to push to git; for information on current and previous contributors, see the [http://www.postgresql.org/community/contributors/ contributor profiles] section of the web site. Note: The names are listed here in order of first commit or when added as a committer, oldest first; this isn't intended to imply anything about depth of contribution.

* Bruce Momjian (momjian)
* Tom Lane (tgl)
* Michael Meskes (meskes)
* Tatsuo Ishii (ishii)
* Peter Eisentraut (petere)
* Joe Conway (joe)
* Alvaro Herrera (alvherre)
* Andrew Dunstan (adunstan)
* Magnus Hagander (mha)
* Heikki Linnakangas (heikki)
* Robert Haas (rhaas)
* Jeff Davis (jdavis)
* Fujii Masao (fujii)
* Noah Misch (noah)
* Andres Freund (andres)
* Dean Rasheed (deanr)
* Andrew Gierth (rhodiumtoad)
* Alexander Korotkov (smagen)
* Amit Kapila (amitkapila)
* Tomas Vondra (fuzzycz)
* Michael Paquier (michael-kun)
* Thomas Munro (macdice)
* Peter Geoghegan (pgeoghegan)
* Etsuro Fujita (efujita)
* David Rowley (drowley)
* Daniel Gustafsson (d_gustafsson)
* John Naylor (john.naylor)
* Nathan Bossart (nathan)
* Amit Langote (amitlan)
* Masahiko Sawada (msawada)

== Notes on the Commit Log ==

Hundreds of developers have successfully contributed work to PostgreSQL over more than 20 years, many acting as individuals, though also many representing academic institutions and both user and vendor companies. Both the "Author" and "Committer" fields of such patches will reflect the committer. The actual author of a patch, if different, is generally listed in the commit message; reviewers or others who contributed ideas or otherwise helped with the patch may also be listed. Many patches, in the form in which they are committed, are the work of multiple people: original author or authors, reviewer(s), and/or committer. As a result, no simple analysis of duration or depth of contribution over time is possible from the commit log. The project operates a system of careful peer review and even committers have their work checked by other committers and the community as a whole.

== New Committers and Removing Committers ==

New committers are added approximately annually by the Core Team. Contributors to PostgreSQL are selected to be committers based on the following loose criteria:

* several years of substantial contributions to the project
* multiple and continuing code contributions
* responsibility for maintenance of one or more areas of the codebase
* track record of reviewing and helping other contributors with their patches
* high quality code contributions which require very little revision or correction for commit
* demonstrated understanding of the process and criteria for patch acceptance

Generally, new committers are selected March or April and announced
at [https://pgconf.dev PGConf.dev] (and historically its predecessor
[https://PGCon PGCon]).

Committers who have become inactive and have not contributed significantly to the PostgreSQL project in several years are removed as committers. Again, the review process for this is approximately annual.

[[Category:Community]]

Committers

2024-02-29T14:49:22Z

Dpage:

This is the current list of people with access to push to the git repository with their user names. For technical details on how committing works, see [[Committing with Git]]. Note: This is just a list of people who currently have access to push to git; for information on current and previous contributors, see the [http://www.postgresql.org/community/contributors/ contributor profiles] section of the web site. Note: The names are listed here in order of first commit or when added as a committer, oldest first; this isn't intended to imply anything about depth of contribution.

* Bruce Momjian (momjian)
* Tom Lane (tgl)
* Michael Meskes (meskes)
* Tatsuo Ishii (ishii)
* Peter Eisentraut (petere)
* Joe Conway (joe)
* Alvaro Herrera (alvherre)
* Andrew Dunstan (adunstan)
* Magnus Hagander (mha)
* Heikki Linnakangas (heikki)
* Robert Haas (rhaas)
* Jeff Davis (jdavis)
* Fujii Masao (fujii)
* Noah Misch (noah)
* Andres Freund (andres)
* Dean Rasheed (deanr)
* Andrew Gierth (rhodiumtoad)
* Alexander Korotkov (smagen)
* Amit Kapila (amitkapila)
* Tomas Vondra (fuzzycz)
* Michael Paquier (michael-kun)
* Thomas Munro (macdice)
* Peter Geoghegan (pgeoghegan)
* Etsuro Fujita (efujita)
* David Rowley (drowley)
* Daniel Gustafsson (d_gustafsson)
* John Naylor (john.naylor)
* Nathan Bossart (nathan)
* Amit Langote (amitlan)
* Masahiko Sawada (msawada)

== Notes on the Commit Log ==

Hundreds of developers have successfully contributed work to PostgreSQL over more than 20 years, many acting as individuals, though also many representing academic institutions and both user and vendor companies. Both the "Author" and "Committer" fields of such patches will reflect the committer. The actual author of a patch, if different, is generally listed in the commit message; reviewers or others who contributed ideas or otherwise helped with the patch may also be listed. Many patches, in the form in which they are committed, are the work of multiple people: original author or authors, reviewer(s), and/or committer. As a result, no simple analysis of duration or depth of contribution over time is possible from the commit log. The project operates a system of careful peer review and even committers have their work checked by other committers and the community as a whole.

== New Committers and Removing Committers ==

New committers are added approximately annually by the Core Team. Contributors to PostgreSQL are selected to be committers based on the following loose criteria:

* several years of substantial contributions to the project
* multiple and continuing code contributions
* responsibility for maintenance of one or more areas of the codebase
* track record of reviewing and helping other contributors with their patches
* high quality code contributions which require very little revision or correction for commit
* demonstrated understanding of the process and criteria for patch acceptance

Generally, new committers are selected March or April and announced at pgCon.

Committers who have become inactive and have not contributed significantly to the PostgreSQL project in several years are removed as committers. Again, the review process for this is approximately annual.

[[Category:Community]]

Handling Security Issues

2024-02-29T14:48:07Z

Dpage: /* Security team */

This page contains information for developers on how to deal with security issues and how to prepare a security release.

'''Note:''' If you are a user who wants to report a security issue or who wants to learn about how the PostgreSQL project deals with security issues, please go to http://www.postgresql.org/support/security. The wiki page you are looking at is aimed at developers.

== security@postgresql.org ==

The email address security@postgresql.org is the recommended contact point for all inquiries about security issues. It currently points at pgsql-security@postgresql.org.

== Security team ==

The security team is a closed-subscription list whose purpose is to be able to discuss security issues in a controlled group. The current members are:

* Álvaro Herrera
* Andres Freund
* Andrew Dunstan
* Bruce Momjian
* Dave Page
* Greg Stark
* Heikki Linnakangas
* Joe Conway
* Jonathan Katz
* Magnus Hagander
* Michael Paquier
* Noah Misch
* Peter Eisentraut
* Robert Haas
* Stefan Kaltenbrunner
* Tom Lane

== Responding to security bug reports ==

Respond to a security bug report reported via the security@ address in the same way as you would with a normal bug report on pgsql-bugs. The only difference is that the issue should not be disclosed to anyone outside the security team and the original reporter.

== Committing patches for security issues ==

If the git commit log message for a commit matches

/^Security:/

then the message on pgsql-committers will be held for moderation. This facility should be applied when committing a patch for a security issue, so that details of the fix don't get broadcast to the world before the release including the fix is made. See [http://archives.postgresql.org/pgsql-committers/2008-01/msg00100.php this example] for a commit message where this was done. As in that example, the recommended style is something like "Security: CVE-number" or some other reference to the reason for the security marker.

== CVE numbers ==

A CVE number should be requested for every security issue. This number should be included in all the commit messages, announcements, and so on. A document describing the process can be found on developer.postgresql.org at ~petere/can-request.txt (only available with shell access, because it is not a public document). The person who drives the process of fixing the issue and preparing the security release (often Tom Lane) is usually the one who deals with requesting the CVE numbers.

[[Category:Development]]

Infrastructure team

2024-02-29T14:32:17Z

Dpage: /* Infrastructure Team Members */

== Infrastructure Team ==

The Infrastructure team is responsible for running all of the postgresql.org infrastructure. This includes a wide variety of public and non-public services. PostgreSQL website development is discussed on the [http://archives.postgresql.org/pgsql-www/ pgsql-www mailing list]. Source code for the postgresql.org web site is stored in a public GIT repository, and is available on [http://git.postgresql.org/gitweb/?p=pgweb.git;a=summary]

== Infrastructure Team Members ==

The following people are part of the sysadmin team running the postgresql.org infrastructure.

* Magnus Hagander
* Álvaro Herrera
* Stefan Kaltenbrunner
* Dave Page
* Joe Conway
* Christophe Pettus

In case you want to reach out to the Infrastructure team, please contact us at sysadmins@lists.postgresql.org

[[Category:Community]]

Infrastructure team

2024-02-29T14:32:06Z

Dpage: /* Infrastructure Team Members */

FOSDEM/PGDay 2024 Developer Meeting

2024-02-01T11:06:15Z

Dpage: /* FOSDEM 2024 Developer Meeting schedule by Time and Room */

== FOSDEM 2024 Developer Meeting schedule by Time and Room ==
{| border="1" cellpadding="4" cellspacing="0"
!Time
!Studio 4

|- style="background-color:lightgray;"
|Thur 9:00-9:10
|Welcome and Introductions

|-
|Thur 9:10-9:30
|Property Graph Queries (Vik)

|-
|Thur 9:30-10:00
|Binary format output per session (Dave C)

|- style="background-color:lightgray;"
|Thur 10:00
|Coffee

|-
|Thur 10:30-11:00
|Meson and packaging (Devrim)

|-
|Thur 11:00-11:30
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)

|-
|Thur 11:30-12:00
|CREATE SUBSCRIPTION ... SERVER (Jeff)

|-
|Thur 12:00-12:30
|The Path to un-reverting the MAINTAIN privilege (Nathan)

|- style="background-color:lightgray;"
|Thur 12:30-13:30
|Lunch

|-
|Thur 13:30-13:40
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").

Note from Daniel: "This was proposed by a hacker who was unable to attend".

|-
|Thur 13:40-13:50
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors.").

Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."

|- style="background-color:white;"
|Thur 13:50-14:00
|Thomas' other thing.
|

|- style="background-color:white;"
|Thur 14:00-14:30
|Page Features / Reserve space on page
|

|- style="background-color:white;"
|Thur 14:30-15:00
|TDE / IVs / More
|

|- style="background-color:lightgray;"
|Thur 15:00
|Tea

|- style="background-color:white;"
|Thur 15:30-16:45
|v17 Patch Triage
|

|- style="background-color:white;"
|Thur 16:45-17:00
|Any other business & close.
|}

== Suggested topics ==

* Patch Triage
* Moving Forward with Pending Patches
* Recognizing New Contributors
* What's new in SQL:2023
* Binary format output per session
* Meson and packaging
* Developer Meeting during PGConf Europe
* Page Features / Reserve space on page
* TDE

== Notes/Minutes ==

=== Property Graph Queries ===

Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.

Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.

Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.

=== Binary format output per session ===

Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.

Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.

Dave C - Don't see being able to extend the protocol?

Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.

Dave C - That's a challenge but the benefit is universal and quite large.

Peter E - Could just make this a connection option.

Dave C - That would be ok too.

Matthias - No type that doesn't have binary.

Peter E - Most do but some extensions may not.

Matthias - When we flush the datum it's binary

Peter E - No, it's the send/recv functions that we're talking about, not the internal.

Dave C - Current default is we send everything as text.

Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..

Peter E - Maybe just have a flag that says everything comes back as binary?

Dave C - Would work for me.

Alvaro - What about pgbouncer?

Peter E - Would still have to track it but would have only one flag to track.

Magnus - Maybe have to have separate pools, one for binary one for text.

Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?

Dave C - Yes, sometimes it does use simple queries.

Alvaro - Why use simple query?

Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).

Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.

Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.

Matthias - How does the driver know about the new type? Register it?

Peter E - Trying to get away from having to register it.

Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.

Matthias - At least now you only use the binary types if you know you can handle it.

Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.

Magnus - Might have to make it optional in the driver if you're using an unknown extension.

Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.

Alvaro - New developer adds new table with query and the app breaks or gets much slower?

Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.

Munro - What about OIDs being reused? No validation machinery.

Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.

Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.

Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)

Munro - Maybe some kind of number assigning authority..

Matthias - Redo manager has a wiki page for extensions to use.

Magnus - Works because there's very few users who need those.

Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.

Dave C - How old are those things?

Joe - Maybe client that ships with RHEL 7 or something?

Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.

Dave C - Have thought about larger protocol changes

Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).

Dave C - v16 is the first version that really handles protocol changes?

Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.

Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.

Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.

Magnus - libpq needs to not fail with this is the main thing.

Dave C - Have some research to do.

=== Meson and packaging ===

Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.

Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.

Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.

Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?

Peter E - Maybe switch early in v18 cycle and try to fix it up.

Berg - As long as it is supposed to work then it's about fixing bugs.

Peter E - Not worthwhile since llvm support just doesn't work at all right now.

Alvaro - Can we insist on disallowing cross-compilation?

Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?

Peter E - People can send to packagers ... and maybe CC other lists

Magnus - Gets weird CC'ing between lists that are private and not private

Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.

Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.

Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.

Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?

Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.

Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.

Munro - Perhaps there needs to be a new document.

Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.

Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.

Bruce - Probably would be best as a separate document.

Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.

Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.

Jeff - Is the biggest issue the dependencies?

Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.

Matthias - Minor release notes have things that are very detailed.

Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.

Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.

Peter E - Would never finish the release notes to include all those.

Stephen - Extension authors should be following hackers or committers if they're using internal structures.

Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.

Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?

Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.

Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.

Matthias - When is there going to be a good guideline on how to build extensions with meson?

Peter E - In the fullness of time.

Matthias - Have that for make but don't see that for meson.

Peter E - No need to actually do that really.

Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.

Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.

Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.

Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.

Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.

=== Built-in collation provider for "C" and "C.UTF-8" ===

Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.

Peter E - What is the difference between this and the C.UTF-8 locale?

Jeff - The C.UTF-8 has changed the collation before, for one thing.

Peter E - Example?

Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.

Peter E - More of a bootstrap problem?

Munro - Yeah, may be able to just ignore that issue as just being a historical problem.

Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.

Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.

Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.

Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.

Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.

Peter E - Who is actually using that?

Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.

Matthias - If I don't care about natural ordering but just care about seeing similar things together.

Peter E - Maybe as a DBA but probably not the case as an application developer.

Matthias - I don't really care about real collation in my apps generally.

Berg - If you're running a server for the world then perhaps it's fine.

Peter E - Maybe we do this for v1 but in v2 add in full support.

Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.

Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.

Munro - A good number of different things are just symlinks between each other.

Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.

Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.

Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.

Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.

Matthias - Recently ICU had a release where they changed but they didn't change the identifier.

Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.

Munro - All those other ones suck though and they're poorly understood.

Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.

Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?

Matthias - Using the GROUP BY with an ORDER BY might be much slower then

Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.

Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.

Matthias - Original proposal was to make primary key text indexes use this

Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.

Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.

Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.

Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.

Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.

Jeff - Not quite type or what the user is going to do with it..

Munro - User could say 'collate C'..

Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.

Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.

Jeff - Database-level collation must be honored and so we would not be changing that.

Vondra - What thought the problem was is that there was a misunderstanding around that.

Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.

Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.

Peter E - Maybe use binary or something instead of C

Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.

Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.

Peter E - I've now signed up as a reviewer...

=== CREATE SUBSCRIPTION ... SERVER ===

Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.

Peter E - Proposed something similar before and therefore generally happy with it.

=== The Path to un-reverting the MAINTAIN privilege ===

=== Moving Forward with Pending Patches ===

(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").

Note from Daniel: "This was proposed by a hacker who was unable to attend".

=== Recognizing New Contributors ===

(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors.").

Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."

=== Page Features / Reserve space on page ===

=== TDE / IVs / More ===

=== v17 Patch Triage ===

=== Any other business & close ===

[[Category:Developer Meeting]]

FOSDEM/PGDay 2024 Developer Meeting

2024-02-01T11:03:55Z

Dpage: /* FOSDEM 2024 Developer Meeting schedule by Time and Room */

== FOSDEM 2024 Developer Meeting schedule by Time and Room ==
{| border="1" cellpadding="4" cellspacing="0"
!Time
!Studio 4

|- style="background-color:lightgray;"
|Thur 9:00-9:10
|Welcome and Introductions

|-
|Thur 9:10-9:30
|Property Graph Queries (Vik)

|-
|Thur 9:30-10:00
|Binary format output per session (Dave C)

|- style="background-color:lightgray;"
|Thur 10:00
|Coffee

|-
|Thur 10:30-11:00
|Meson and packaging (Devrim)

|-
|Thur 11:00-11:30
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)

|-
|Thur 11:30-12:00
|CREATE SUBSCRIPTION ... SERVER (Jeff)

|-
|Thur 12:00-12:30
|The Path to un-reverting the MAINTAIN privilege (Nathan)

|- style="background-color:lightgray;"
|Thur 12:30-13:30
|Lunch

|-
|Thur 13:30-13:45
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").

Note from Daniel: "This was proposed by a hacker who was unable to attend".

|-
|Thur 13:45-14:00
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors.").

Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."

|- style="background-color:white;"
|Thur 14:00-14:15
|Page Features / Reserve space on page
|

|- style="background-color:white;"
|Thur 14:15-14:30
|Thomas' other thing.
|

|- style="background-color:white;"
|Thur 14:30-15:00
|TDE / IVs / More
|

|- style="background-color:lightgray;"
|Thur 15:00
|Tea

|- style="background-color:white;"
|Thur 15:30-16:45
|v17 Patch Triage
|

|- style="background-color:white;"
|Thur 16:45-17:00
|Any other business & close.
|}

== Suggested topics ==

* Patch Triage
* Moving Forward with Pending Patches
* Recognizing New Contributors
* What's new in SQL:2023
* Binary format output per session
* Meson and packaging
* Developer Meeting during PGConf Europe
* Page Features / Reserve space on page
* TDE

== Notes/Minutes ==

=== Property Graph Queries ===

Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.

Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.

Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.

=== Binary format output per session ===

Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.

Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.

Dave C - Don't see being able to extend the protocol?

Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.

Dave C - That's a challenge but the benefit is universal and quite large.

Peter E - Could just make this a connection option.

Dave C - That would be ok too.

Matthias - No type that doesn't have binary.

Peter E - Most do but some extensions may not.

Matthias - When we flush the datum it's binary

Peter E - No, it's the send/recv functions that we're talking about, not the internal.

Dave C - Current default is we send everything as text.

Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..

Peter E - Maybe just have a flag that says everything comes back as binary?

Dave C - Would work for me.

Alvaro - What about pgbouncer?

Peter E - Would still have to track it but would have only one flag to track.

Magnus - Maybe have to have separate pools, one for binary one for text.

Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?

Dave C - Yes, sometimes it does use simple queries.

Alvaro - Why use simple query?

Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).

Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.

Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.

Matthias - How does the driver know about the new type? Register it?

Peter E - Trying to get away from having to register it.

Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.

Matthias - At least now you only use the binary types if you know you can handle it.

Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.

Magnus - Might have to make it optional in the driver if you're using an unknown extension.

Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.

Alvaro - New developer adds new table with query and the app breaks or gets much slower?

Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.

Munro - What about OIDs being reused? No validation machinery.

Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.

Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.

Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)

Munro - Maybe some kind of number assigning authority..

Matthias - Redo manager has a wiki page for extensions to use.

Magnus - Works because there's very few users who need those.

Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.

Dave C - How old are those things?

Joe - Maybe client that ships with RHEL 7 or something?

Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.

Dave C - Have thought about larger protocol changes

Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).

Dave C - v16 is the first version that really handles protocol changes?

Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.

Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.

Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.

Magnus - libpq needs to not fail with this is the main thing.

Dave C - Have some research to do.

=== Meson and packaging ===

Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.

Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.

Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.

Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?

Peter E - Maybe switch early in v18 cycle and try to fix it up.

Berg - As long as it is supposed to work then it's about fixing bugs.

Peter E - Not worthwhile since llvm support just doesn't work at all right now.

Alvaro - Can we insist on disallowing cross-compilation?

Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?

Peter E - People can send to packagers ... and maybe CC other lists

Magnus - Gets weird CC'ing between lists that are private and not private

Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.

Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.

Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.

Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?

Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.

Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.

Munro - Perhaps there needs to be a new document.

Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.

Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.

Bruce - Probably would be best as a separate document.

Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.

Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.

Jeff - Is the biggest issue the dependencies?

Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.

Matthias - Minor release notes have things that are very detailed.

Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.

Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.

Peter E - Would never finish the release notes to include all those.

Stephen - Extension authors should be following hackers or committers if they're using internal structures.

Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.

Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?

Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.

Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.

Matthias - When is there going to be a good guideline on how to build extensions with meson?

Peter E - In the fullness of time.

Matthias - Have that for make but don't see that for meson.

Peter E - No need to actually do that really.

Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.

Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.

Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.

Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.

Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.

=== Built-in collation provider for "C" and "C.UTF-8" ===

Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.

Peter E - What is the difference between this and the C.UTF-8 locale?

Jeff - The C.UTF-8 has changed the collation before, for one thing.

Peter E - Example?

Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.

Peter E - More of a bootstrap problem?

Munro - Yeah, may be able to just ignore that issue as just being a historical problem.

Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.

Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.

Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.

Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.

Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.

Peter E - Who is actually using that?

Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.

Matthias - If I don't care about natural ordering but just care about seeing similar things together.

Peter E - Maybe as a DBA but probably not the case as an application developer.

Matthias - I don't really care about real collation in my apps generally.

Berg - If you're running a server for the world then perhaps it's fine.

Peter E - Maybe we do this for v1 but in v2 add in full support.

Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.

Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.

Munro - A good number of different things are just symlinks between each other.

Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.

Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.

Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.

Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.

Matthias - Recently ICU had a release where they changed but they didn't change the identifier.

Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.

Munro - All those other ones suck though and they're poorly understood.

Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.

Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?

Matthias - Using the GROUP BY with an ORDER BY might be much slower then

Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.

Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.

Matthias - Original proposal was to make primary key text indexes use this

Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.

Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.

Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.

Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.

Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.

Jeff - Not quite type or what the user is going to do with it..

Munro - User could say 'collate C'..

Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.

Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.

Jeff - Database-level collation must be honored and so we would not be changing that.

Vondra - What thought the problem was is that there was a misunderstanding around that.

Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.

Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.

Peter E - Maybe use binary or something instead of C

Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.

Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.

Peter E - I've now signed up as a reviewer...

=== CREATE SUBSCRIPTION ... SERVER ===

Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.

Peter E - Proposed something similar before and therefore generally happy with it.

=== The Path to un-reverting the MAINTAIN privilege ===

=== Moving Forward with Pending Patches ===

(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").

Note from Daniel: "This was proposed by a hacker who was unable to attend".

=== Recognizing New Contributors ===

(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors.").

Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."

=== Page Features / Reserve space on page ===

=== TDE / IVs / More ===

=== v17 Patch Triage ===

=== Any other business & close ===

[[Category:Developer Meeting]]

FOSDEM/PGDay 2024 Developer Meeting

2024-02-01T11:01:37Z

Dpage: /* FOSDEM 2024 Developer Meeting schedule by Time and Room */

== FOSDEM 2024 Developer Meeting schedule by Time and Room ==
{| border="1" cellpadding="4" cellspacing="0"
!Time
!Studio 4

|- style="background-color:lightgray;"
|Thur 9:00-9:10
|Welcome and Introductions

|-
|Thur 9:10-9:30
|Property Graph Queries (Vik)

|-
|Thur 9:30-10:00
|Binary format output per session (Dave C)

|- style="background-color:lightgray;"
|Thur 10:00
|Coffee

|-
|Thur 10:30-11:00
|Meson and packaging (Devrim)

|-
|Thur 11:00-11:30
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)

|-
|Thur 11:30-12:00
|CREATE SUBSCRIPTION ... SERVER (Jeff)

|-
|Thur 12:00-12:30
|The Path to un-reverting the MAINTAIN privilege (Nathan)

|- style="background-color:lightgray;"
|Thur 12:30-13:30
|Lunch

|-
|Thur 13:30-13:45
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").

Note from Daniel: "This was proposed by a hacker who was unable to attend".

|-
|Thur 13:45-14:00
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors.").

Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."

|- style="background-color:white;"
|Thur 14:00-14:30
|Page Features / Reserve space on page
|

|- style="background-color:white;"
|Thur 14:30-15:00
|TDE / IVs / More
|

|- style="background-color:lightgray;"
|Thur 15:00
|Tea

|- style="background-color:white;"
|Thur 15:30-16:45
|v17 Patch Triage
|

|- style="background-color:white;"
|Thur 16:45-17:00
|Any other business & close.
|}

== Suggested topics ==

* Patch Triage
* Moving Forward with Pending Patches
* Recognizing New Contributors
* What's new in SQL:2023
* Binary format output per session
* Meson and packaging
* Developer Meeting during PGConf Europe
* Page Features / Reserve space on page
* TDE

== Notes/Minutes ==

=== Property Graph Queries ===

Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.

Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.

Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.

=== Binary format output per session ===

Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.

Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.

Dave C - Don't see being able to extend the protocol?

Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.

Dave C - That's a challenge but the benefit is universal and quite large.

Peter E - Could just make this a connection option.

Dave C - That would be ok too.

Matthias - No type that doesn't have binary.

Peter E - Most do but some extensions may not.

Matthias - When we flush the datum it's binary

Peter E - No, it's the send/recv functions that we're talking about, not the internal.

Dave C - Current default is we send everything as text.

Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..

Peter E - Maybe just have a flag that says everything comes back as binary?

Dave C - Would work for me.

Alvaro - What about pgbouncer?

Peter E - Would still have to track it but would have only one flag to track.

Magnus - Maybe have to have separate pools, one for binary one for text.

Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?

Dave C - Yes, sometimes it does use simple queries.

Alvaro - Why use simple query?

Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).

Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.

Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.

Matthias - How does the driver know about the new type? Register it?

Peter E - Trying to get away from having to register it.

Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.

Matthias - At least now you only use the binary types if you know you can handle it.

Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.

Magnus - Might have to make it optional in the driver if you're using an unknown extension.

Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.

Alvaro - New developer adds new table with query and the app breaks or gets much slower?

Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.

Munro - What about OIDs being reused? No validation machinery.

Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.

Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.

Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)

Munro - Maybe some kind of number assigning authority..

Matthias - Redo manager has a wiki page for extensions to use.

Magnus - Works because there's very few users who need those.

Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.

Dave C - How old are those things?

Joe - Maybe client that ships with RHEL 7 or something?

Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.

Dave C - Have thought about larger protocol changes

Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).

Dave C - v16 is the first version that really handles protocol changes?

Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.

Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.

Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.

Magnus - libpq needs to not fail with this is the main thing.

Dave C - Have some research to do.

=== Meson and packaging ===

Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.

Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.

Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.

Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?

Peter E - Maybe switch early in v18 cycle and try to fix it up.

Berg - As long as it is supposed to work then it's about fixing bugs.

Peter E - Not worthwhile since llvm support just doesn't work at all right now.

Alvaro - Can we insist on disallowing cross-compilation?

Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?

Peter E - People can send to packagers ... and maybe CC other lists

Magnus - Gets weird CC'ing between lists that are private and not private

Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.

Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.

Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.

Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?

Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.

Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.

Munro - Perhaps there needs to be a new document.

Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.

Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.

Bruce - Probably would be best as a separate document.

Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.

Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.

Jeff - Is the biggest issue the dependencies?

Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.

Matthias - Minor release notes have things that are very detailed.

Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.

Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.

Peter E - Would never finish the release notes to include all those.

Stephen - Extension authors should be following hackers or committers if they're using internal structures.

Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.

Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?

Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.

Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.

Matthias - When is there going to be a good guideline on how to build extensions with meson?

Peter E - In the fullness of time.

Matthias - Have that for make but don't see that for meson.

Peter E - No need to actually do that really.

Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.

Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.

Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.

Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.

Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.

=== Built-in collation provider for "C" and "C.UTF-8" ===

Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.

Peter E - What is the difference between this and the C.UTF-8 locale?

Jeff - The C.UTF-8 has changed the collation before, for one thing.

Peter E - Example?

Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.

Peter E - More of a bootstrap problem?

Munro - Yeah, may be able to just ignore that issue as just being a historical problem.

Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.

Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.

Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.

Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.

Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.

Peter E - Who is actually using that?

Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.

Matthias - If I don't care about natural ordering but just care about seeing similar things together.

Peter E - Maybe as a DBA but probably not the case as an application developer.

Matthias - I don't really care about real collation in my apps generally.

Berg - If you're running a server for the world then perhaps it's fine.

Peter E - Maybe we do this for v1 but in v2 add in full support.

Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.

Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.

Munro - A good number of different things are just symlinks between each other.

Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.

Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.

Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.

Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.

Matthias - Recently ICU had a release where they changed but they didn't change the identifier.

Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.

Munro - All those other ones suck though and they're poorly understood.

Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.

Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?

Matthias - Using the GROUP BY with an ORDER BY might be much slower then

Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.

Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.

Matthias - Original proposal was to make primary key text indexes use this

Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.

Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.

Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.

Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.

Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.

Jeff - Not quite type or what the user is going to do with it..

Munro - User could say 'collate C'..

Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.

Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.

Jeff - Database-level collation must be honored and so we would not be changing that.

Vondra - What thought the problem was is that there was a misunderstanding around that.

Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.

Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.

Peter E - Maybe use binary or something instead of C

Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.

Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.

Peter E - I've now signed up as a reviewer...

=== CREATE SUBSCRIPTION ... SERVER ===

Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.

Peter E - Proposed something similar before and therefore generally happy with it.

=== The Path to un-reverting the MAINTAIN privilege ===

=== Moving Forward with Pending Patches ===

(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").

Note from Daniel: "This was proposed by a hacker who was unable to attend".

=== Recognizing New Contributors ===

(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors.").

Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."

=== Page Features / Reserve space on page ===

=== TDE / IVs / More ===

=== v17 Patch Triage ===

=== Any other business & close ===

[[Category:Developer Meeting]]

2023-05-26T10:52:24Z

Dpage:

PgCon 2023 Developer Meeting

2023-03-21T16:08:41Z

Dpage:

2019-04-16T07:49:04Z

Dpage: /* Agenda */

A meeting of the interested PostgreSQL developers is being planned for Tuesday 28 May, 2019 at the University of Ottawa, prior to pgCon 2019. In order to keep the numbers manageable, this meeting is by '''invitation only'''.

The invitation list for the meeting has changed this year to include representatives from various project sub-teams, for example, packagers, the release team, Code of Conduct committee and more.

As at last years event, an Unconference will be held on Wednesday for in-depth discussion of technical topics.

This is a PostgreSQL Community event.

== Meeting Goals ==

* Define the schedule for the 13.0 release cycle
* Address any proposed timing, policy, or procedure issues
* Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]

== Time & Location ==

The meeting will be:

* 9:00AM to 12PM
* DMS TBC
* University of Ottawa.

Coffee, tea and snacks will be served starting at 8:45am. Lunch will be after the meeting.

== RSVPs ==

The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!

# Joe Conway
# Jeff Davis
# Andrew Dunstan
# Peter Eisentraut
# Andres Freund
# Stephen Frost
# Peter Geoghegan
# Devrim Gündüz
# Robert Haas
# Magnus Hagander
# Stacey Haysler (present for CoC presentation and discussion only)
# Álvaro Herrera
# Amit Kapila
# Jonathan Katz
# Alexander Korotkov
# Tom Lane
# Heikki Linnakangas
# Noah Misch
# Bruce Momjian
# Thomas Munro
# Dave Page
# Michael Paquier
# Tomas Vondra

The following people will not be in Ottawa, and do not plan to attend:

* Christoph Berg
* Andreas Scherbaum
* Sarah Conway Schnurr
* Ilya Kosmodemiansky

== Agenda Items ==

* 13.0 release and commitfest schedule (Dave)
* Contributor Recognition (Andres, happy to share / pass, but should be discussed)
** [https://www.postgresql.org/community/contributors/ contributors] page update - how well is it working?
** should the developer meeting serve as recognition?
* The Evolving Developer Meeting: Goals & What We Want to Accomplish by the end of it? (Jonathan, happy to share)
* SQL standard update (Peter E.)
* ''Please add suggestions for agenda items here. (with your name)''

==Agenda==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter

|- style="font-style:italic;background-color:lightgray;"
|09:00 - 09:10
|Welcome and introductions
|Dave Page

|-
|09:10 - 09:30
|Code of Conduct report and discussion
|Stacey Haysler

|-
|09:30 - 09:45
|12.0 release and commitfest schedule
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|10:30 - 11:00
|Coffee break
|All

|-
|11:50 - 12:00
|Any other business
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|12:00
|Lunch
|

|}

Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move however). Any remaining time before lunch may be used for Commitfest item triage or other activities.

PgCon 2019 Developer Meeting

2019-04-16T07:47:49Z

Dpage: /* RSVPs */

A meeting of the interested PostgreSQL developers is being planned for Tuesday 28 May, 2019 at the University of Ottawa, prior to pgCon 2019. In order to keep the numbers manageable, this meeting is by '''invitation only'''.

The invitation list for the meeting has changed this year to include representatives from various project sub-teams, for example, packagers, the release team, Code of Conduct committee and more.

As at last years event, an Unconference will be held on Wednesday for in-depth discussion of technical topics.

This is a PostgreSQL Community event.

== Meeting Goals ==

* Define the schedule for the 13.0 release cycle
* Address any proposed timing, policy, or procedure issues
* Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]

== Time & Location ==

The meeting will be:

* 9:00AM to 12PM
* DMS TBC
* University of Ottawa.

Coffee, tea and snacks will be served starting at 8:45am. Lunch will be after the meeting.

== RSVPs ==

The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!

# Joe Conway
# Jeff Davis
# Andrew Dunstan
# Peter Eisentraut
# Andres Freund
# Stephen Frost
# Peter Geoghegan
# Devrim Gündüz
# Robert Haas
# Magnus Hagander
# Stacey Haysler (present for CoC presentation and discussion only)
# Álvaro Herrera
# Amit Kapila
# Jonathan Katz
# Alexander Korotkov
# Tom Lane
# Heikki Linnakangas
# Noah Misch
# Bruce Momjian
# Thomas Munro
# Dave Page
# Michael Paquier
# Tomas Vondra

The following people will not be in Ottawa, and do not plan to attend:

* Christoph Berg
* Andreas Scherbaum
* Sarah Conway Schnurr
* Ilya Kosmodemiansky

== Agenda Items ==

* 13.0 release and commitfest schedule (Dave)
* Contributor Recognition (Andres, happy to share / pass, but should be discussed)
** [https://www.postgresql.org/community/contributors/ contributors] page update - how well is it working?
** should the developer meeting serve as recognition?
* The Evolving Developer Meeting: Goals & What We Want to Accomplish by the end of it? (Jonathan, happy to share)
* SQL standard update (Peter E.)
* ''Please add suggestions for agenda items here. (with your name)''

==Agenda==

{| border="1" cellpadding="4" cellspacing="0"
!Time
!Item
!Presenter

|- style="font-style:italic;background-color:lightgray;"
|09:00 - 09:30
|Welcome and introductions
|Dave Page

|-
|09:30 - 09:45
|12.0 release and commitfest schedule
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|10:30 - 11:00
|Coffee break
|All

|-
|11:50 - 12:00
|Any other business
|Dave Page

|- style="font-style:italic;background-color:lightgray;"
|12:00
|Lunch
|

|}

Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move however). Any remaining time before lunch may be used for Commitfest item triage or other activities.

PgCon 2019 Developer Meeting

2019-04-15T08:05:00Z

Dpage: