https://wiki.postgresql.org/api.php?action=feedcontributions&user=Sfrost&feedformat=atomPostgreSQL wiki - User contributions [en]2024-03-29T10:43:31ZUser contributionsMediaWiki 1.35.13https://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38662FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:54:22Z<p>Sfrost: </p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
<br />
Jeff - Might make it if given some help<br />
<br />
Vondra - Concern on that patch is that the selectivity on range types will be better than on range conditions which seems bizarre. Not a good reason to reject it though.<br />
<br />
=== logical decoding and replication of sequences, take 2 ===<br />
<br />
Vondra - Lots of discussion about this at pgconf.eu. People are asking if this should be built in to logical replication. You would never use sequences for primary/primary, but for things like failover it makes sense to have this. Alternative solution would be to have a separate tool to handle this. Consensus was that the easier to use the better and easier is to have it be built-in. Desirable to have it be built-in. Performance impact was a concern and this makes logical decoding do more work. In some corner cases like super heavy usage of sequences then this could have a 10x increase in decoding time. Tried to solve by adding a special flag that disables it and avoids the overhead of it. If you need to decode and apply sequences then you can set a flag during subscription creation. When you know you are creating a subscription for upgrade, for example, then you would enable the option and then it would handle it. Doing more work requires time. Question of correctness also raised because sequences are a bit annoying because they are not transactional. When you get things from a sequence, it's immediately visible to other sessions. There's some complexity to deal with that and in thinking about it. During the decoding you have one timeline for the original database and then you have a timeline of the decoding which may happen later and then you have the final timeline in which the changes are applied. This means it isn't the same order as the original and when sequences are not transactional this is causing issues. Some questions about if the way that the sequences are decoded is correct or not. There are maybe some bugs but if there are the question is if they can be fixed or if they are fundamental.<br />
<br />
Jeff - Is there consensus around what correct is in this context?<br />
<br />
Vondra - In principle, the same thing applies to logical replication in general- it should be the same as the commit order on the source database. For example, you shouldn't see a sequence going backwards. You shouldn't see an older version of the sequence.<br />
<br />
Jeff - Would you potentially see spurious updates that didn't make it to the primary? Would that be considered incorrect?<br />
<br />
Vondra - Don't see how you could get that and if it didn't make it into the transaction log then it didn't happen.<br />
<br />
Jeff - Perhaps we can discuss later<br />
<br />
Vondra - Happy to do that. Patch implementation has changed a few times. Eventually went back to the original simple decoding. Not sure how to improve that. Understand the concerns that have been raised on the list around the complexity of the interactions of these different timelines. Changes can happen in different orderings. In principle it might be wrong but that's very hard to prove. Not intent to just commit something to commit it and have to deal with bugs, want to discuss and try to solve them ahead of time.<br />
<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
<br />
Stephen - This continues to be worked on and there is recent activity on it.<br />
<br />
=== Multi-version ICU ===<br />
<br />
Munro - Not going to be in v17<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE ... RETURNING ===<br />
<br />
Jeff - Wanted to bring this up. There are some specifics about which of the keyword lists these keywords are included in. Would be nice to have someone look at them. Would like a sanity check as to which keyword list these keywords are being added to.<br />
<br />
Vik - Haven't seen these keywords specifically. The only place this could be used in a RETURNING clause and so there is no risk of conflict with that...?<br />
<br />
Jeff - Sort of ... There are some questions about the details here. Being held up on linguistic issues. There is a current patch which has answers for those details. Not 100% confident that those are correct. Feels better implementation wise.<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
<br />
Peter E - Lots of interest in having some way to assess if the backend ABI was changed as sometimes that happens by accident. abidiff is a Linux tool which checks if the ABI has changed. Various levels that can be done with it. Wondering if there is interest in this?<br />
<br />
Stephen - Seems like it would be good to have<br />
<br />
Berg - Packagers would probably be helped by this too<br />
<br />
Peter E - Ideally this is something which would be caught before a release goes out with an unintentional change. Lots of options and you can write an exclude file to exclude things which are decided is acceptable.<br />
<br />
Berg - Debian is tracking functions exported by libraries<br />
<br />
Peter E - This also checks libraries. First patch just adds simple targets to generate XML files. Second patch adds a test suite to add it to CI. Some question about if this is stable enough across different platforms.</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38661FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:51:30Z<p>Sfrost: /* MERGE ... RETURNING */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
<br />
Jeff - Might make it if given some help<br />
<br />
Vondra - Concern on that patch is that the selectivity on range types will be better than on range conditions which seems bizarre. Not a good reason to reject it though.<br />
<br />
=== logical decoding and replication of sequences, take 2 ===<br />
<br />
Vondra - Lots of discussion about this at pgconf.eu. People are asking if this should be built in to logical replication. You would never use sequences for primary/primary, but for things like failover it makes sense to have this. Alternative solution would be to have a separate tool to handle this. Consensus was that the easier to use the better and easier is to have it be built-in. Desirable to have it be built-in. Performance impact was a concern and this makes logical decoding do more work. In some corner cases like super heavy usage of sequences then this could have a 10x increase in decoding time. Tried to solve by adding a special flag that disables it and avoids the overhead of it. If you need to decode and apply sequences then you can set a flag during subscription creation. When you know you are creating a subscription for upgrade, for example, then you would enable the option and then it would handle it. Doing more work requires time. Question of correctness also raised because sequences are a bit annoying because they are not transactional. When you get things from a sequence, it's immediately visible to other sessions. There's some complexity to deal with that and in thinking about it. During the decoding you have one timeline for the original database and then you have a timeline of the decoding which may happen later and then you have the final timeline in which the changes are applied. This means it isn't the same order as the original and when sequences are not transactional this is causing issues. Some questions about if the way that the sequences are decoded is correct or not. There are maybe some bugs but if there are the question is if they can be fixed or if they are fundamental.<br />
<br />
Jeff - Is there consensus around what correct is in this context?<br />
<br />
Vondra - In principle, the same thing applies to logical replication in general- it should be the same as the commit order on the source database. For example, you shouldn't see a sequence going backwards. You shouldn't see an older version of the sequence.<br />
<br />
Jeff - Would you potentially see spurious updates that didn't make it to the primary? Would that be considered incorrect?<br />
<br />
Vondra - Don't see how you could get that and if it didn't make it into the transaction log then it didn't happen.<br />
<br />
Jeff - Perhaps we can discuss later<br />
<br />
Vondra - Happy to do that. Patch implementation has changed a few times. Eventually went back to the original simple decoding. Not sure how to improve that. Understand the concerns that have been raised on the list around the complexity of the interactions of these different timelines. Changes can happen in different orderings. In principle it might be wrong but that's very hard to prove. Not intent to just commit something to commit it and have to deal with bugs, want to discuss and try to solve them ahead of time.<br />
<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
<br />
Stephen - This continues to be worked on and there is recent activity on it.<br />
<br />
=== Multi-version ICU ===<br />
<br />
Munro - Not going to be in v17<br />
<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
<br />
Jeff - Wanted to bring this up. There are some specifics about which of the keyword lists these keywords are included in. Would be nice to have someone look at them. Would like a sanity check as to which keyword list these keywords are being added to.<br />
<br />
Vik - Haven't seen these keywords specifically. The only place this could be used in a RETURNING clause and so there is no risk of conflict with that...?<br />
<br />
Jeff - Sort of ... There are some questions about the details here. Being held up on linguistic issues. There is a current patch which has answers for those details. Not 100% confident that those are correct. Feels better implementation wise.<br />
<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
<br />
Peter E - Lots of interest in having some way to assess if the backend ABI was changed as sometimes that happens by accident. abidiff is a Linux tool which checks if the ABI has changed. Various levels that can be done with it. Wondering if there is interest in this?<br />
<br />
Stephen - Seems like it would be good to have<br />
<br />
Berg - Packagers would probably be helped by this too<br />
<br />
Peter E - Ideally this is something which would be caught before a release goes out with an unintentional change. Lots of options and you can write an exclude file to exclude things which are decided is acceptable.<br />
<br />
Berg - Debian is tracking functions exported by libraries<br />
<br />
Peter E - This also checks libraries. First patch just adds simple targets to generate XML files. Second patch adds a test suite to add it to CI. Some question about if this is stable enough across different platforms.<br />
<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38660FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:47:40Z<p>Sfrost: /* abidiff tests */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
<br />
Jeff - Might make it if given some help<br />
<br />
Vondra - Concern on that patch is that the selectivity on range types will be better than on range conditions which seems bizarre. Not a good reason to reject it though.<br />
<br />
=== logical decoding and replication of sequences, take 2 ===<br />
<br />
Vondra - Lots of discussion about this at pgconf.eu. People are asking if this should be built in to logical replication. You would never use sequences for primary/primary, but for things like failover it makes sense to have this. Alternative solution would be to have a separate tool to handle this. Consensus was that the easier to use the better and easier is to have it be built-in. Desirable to have it be built-in. Performance impact was a concern and this makes logical decoding do more work. In some corner cases like super heavy usage of sequences then this could have a 10x increase in decoding time. Tried to solve by adding a special flag that disables it and avoids the overhead of it. If you need to decode and apply sequences then you can set a flag during subscription creation. When you know you are creating a subscription for upgrade, for example, then you would enable the option and then it would handle it. Doing more work requires time. Question of correctness also raised because sequences are a bit annoying because they are not transactional. When you get things from a sequence, it's immediately visible to other sessions. There's some complexity to deal with that and in thinking about it. During the decoding you have one timeline for the original database and then you have a timeline of the decoding which may happen later and then you have the final timeline in which the changes are applied. This means it isn't the same order as the original and when sequences are not transactional this is causing issues. Some questions about if the way that the sequences are decoded is correct or not. There are maybe some bugs but if there are the question is if they can be fixed or if they are fundamental.<br />
<br />
Jeff - Is there consensus around what correct is in this context?<br />
<br />
Vondra - In principle, the same thing applies to logical replication in general- it should be the same as the commit order on the source database. For example, you shouldn't see a sequence going backwards. You shouldn't see an older version of the sequence.<br />
<br />
Jeff - Would you potentially see spurious updates that didn't make it to the primary? Would that be considered incorrect?<br />
<br />
Vondra - Don't see how you could get that and if it didn't make it into the transaction log then it didn't happen.<br />
<br />
Jeff - Perhaps we can discuss later<br />
<br />
Vondra - Happy to do that. Patch implementation has changed a few times. Eventually went back to the original simple decoding. Not sure how to improve that. Understand the concerns that have been raised on the list around the complexity of the interactions of these different timelines. Changes can happen in different orderings. In principle it might be wrong but that's very hard to prove. Not intent to just commit something to commit it and have to deal with bugs, want to discuss and try to solve them ahead of time.<br />
<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
<br />
Stephen - This continues to be worked on and there is recent activity on it.<br />
<br />
=== Multi-version ICU ===<br />
<br />
Munro - Not going to be in v17<br />
<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
<br />
Peter E - Lots of interest in having some way to assess if the backend ABI was changed as sometimes that happens by accident. abidiff is a Linux tool which checks if the ABI has changed. Various levels that can be done with it. Wondering if there is interest in this?<br />
<br />
Stephen - Seems like it would be good to have<br />
<br />
Berg - Packagers would probably be helped by this too<br />
<br />
Peter E - Ideally this is something which would be caught before a release goes out with an unintentional change. Lots of options and you can write an exclude file to exclude things which are decided is acceptable.<br />
<br />
Berg - Debian is tracking functions exported by libraries<br />
<br />
Peter E - This also checks libraries. First patch just adds simple targets to generate XML files. Second patch adds a test suite to add it to CI. Some question about if this is stable enough across different platforms.<br />
<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38659FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:42:37Z<p>Sfrost: /* Multi-version ICU */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
<br />
Jeff - Might make it if given some help<br />
<br />
Vondra - Concern on that patch is that the selectivity on range types will be better than on range conditions which seems bizarre. Not a good reason to reject it though.<br />
<br />
=== logical decoding and replication of sequences, take 2 ===<br />
<br />
Vondra - Lots of discussion about this at pgconf.eu. People are asking if this should be built in to logical replication. You would never use sequences for primary/primary, but for things like failover it makes sense to have this. Alternative solution would be to have a separate tool to handle this. Consensus was that the easier to use the better and easier is to have it be built-in. Desirable to have it be built-in. Performance impact was a concern and this makes logical decoding do more work. In some corner cases like super heavy usage of sequences then this could have a 10x increase in decoding time. Tried to solve by adding a special flag that disables it and avoids the overhead of it. If you need to decode and apply sequences then you can set a flag during subscription creation. When you know you are creating a subscription for upgrade, for example, then you would enable the option and then it would handle it. Doing more work requires time. Question of correctness also raised because sequences are a bit annoying because they are not transactional. When you get things from a sequence, it's immediately visible to other sessions. There's some complexity to deal with that and in thinking about it. During the decoding you have one timeline for the original database and then you have a timeline of the decoding which may happen later and then you have the final timeline in which the changes are applied. This means it isn't the same order as the original and when sequences are not transactional this is causing issues. Some questions about if the way that the sequences are decoded is correct or not. There are maybe some bugs but if there are the question is if they can be fixed or if they are fundamental.<br />
<br />
Jeff - Is there consensus around what correct is in this context?<br />
<br />
Vondra - In principle, the same thing applies to logical replication in general- it should be the same as the commit order on the source database. For example, you shouldn't see a sequence going backwards. You shouldn't see an older version of the sequence.<br />
<br />
Jeff - Would you potentially see spurious updates that didn't make it to the primary? Would that be considered incorrect?<br />
<br />
Vondra - Don't see how you could get that and if it didn't make it into the transaction log then it didn't happen.<br />
<br />
Jeff - Perhaps we can discuss later<br />
<br />
Vondra - Happy to do that. Patch implementation has changed a few times. Eventually went back to the original simple decoding. Not sure how to improve that. Understand the concerns that have been raised on the list around the complexity of the interactions of these different timelines. Changes can happen in different orderings. In principle it might be wrong but that's very hard to prove. Not intent to just commit something to commit it and have to deal with bugs, want to discuss and try to solve them ahead of time.<br />
<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
<br />
Stephen - This continues to be worked on and there is recent activity on it.<br />
<br />
=== Multi-version ICU ===<br />
<br />
Munro - Not going to be in v17<br />
<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38658FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:42:17Z<p>Sfrost: /* Patch to implement missing join selectivity estimation for range types */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
<br />
Jeff - Might make it if given some help<br />
<br />
Vondra - Concern on that patch is that the selectivity on range types will be better than on range conditions which seems bizarre. Not a good reason to reject it though.<br />
<br />
=== logical decoding and replication of sequences, take 2 ===<br />
<br />
Vondra - Lots of discussion about this at pgconf.eu. People are asking if this should be built in to logical replication. You would never use sequences for primary/primary, but for things like failover it makes sense to have this. Alternative solution would be to have a separate tool to handle this. Consensus was that the easier to use the better and easier is to have it be built-in. Desirable to have it be built-in. Performance impact was a concern and this makes logical decoding do more work. In some corner cases like super heavy usage of sequences then this could have a 10x increase in decoding time. Tried to solve by adding a special flag that disables it and avoids the overhead of it. If you need to decode and apply sequences then you can set a flag during subscription creation. When you know you are creating a subscription for upgrade, for example, then you would enable the option and then it would handle it. Doing more work requires time. Question of correctness also raised because sequences are a bit annoying because they are not transactional. When you get things from a sequence, it's immediately visible to other sessions. There's some complexity to deal with that and in thinking about it. During the decoding you have one timeline for the original database and then you have a timeline of the decoding which may happen later and then you have the final timeline in which the changes are applied. This means it isn't the same order as the original and when sequences are not transactional this is causing issues. Some questions about if the way that the sequences are decoded is correct or not. There are maybe some bugs but if there are the question is if they can be fixed or if they are fundamental.<br />
<br />
Jeff - Is there consensus around what correct is in this context?<br />
<br />
Vondra - In principle, the same thing applies to logical replication in general- it should be the same as the commit order on the source database. For example, you shouldn't see a sequence going backwards. You shouldn't see an older version of the sequence.<br />
<br />
Jeff - Would you potentially see spurious updates that didn't make it to the primary? Would that be considered incorrect?<br />
<br />
Vondra - Don't see how you could get that and if it didn't make it into the transaction log then it didn't happen.<br />
<br />
Jeff - Perhaps we can discuss later<br />
<br />
Vondra - Happy to do that. Patch implementation has changed a few times. Eventually went back to the original simple decoding. Not sure how to improve that. Understand the concerns that have been raised on the list around the complexity of the interactions of these different timelines. Changes can happen in different orderings. In principle it might be wrong but that's very hard to prove. Not intent to just commit something to commit it and have to deal with bugs, want to discuss and try to solve them ahead of time.<br />
<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
<br />
Stephen - This continues to be worked on and there is recent activity on it.<br />
<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38657FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:41:02Z<p>Sfrost: /* logical decoding and replication of sequences, take 2 */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
<br />
Vondra - Lots of discussion about this at pgconf.eu. People are asking if this should be built in to logical replication. You would never use sequences for primary/primary, but for things like failover it makes sense to have this. Alternative solution would be to have a separate tool to handle this. Consensus was that the easier to use the better and easier is to have it be built-in. Desirable to have it be built-in. Performance impact was a concern and this makes logical decoding do more work. In some corner cases like super heavy usage of sequences then this could have a 10x increase in decoding time. Tried to solve by adding a special flag that disables it and avoids the overhead of it. If you need to decode and apply sequences then you can set a flag during subscription creation. When you know you are creating a subscription for upgrade, for example, then you would enable the option and then it would handle it. Doing more work requires time. Question of correctness also raised because sequences are a bit annoying because they are not transactional. When you get things from a sequence, it's immediately visible to other sessions. There's some complexity to deal with that and in thinking about it. During the decoding you have one timeline for the original database and then you have a timeline of the decoding which may happen later and then you have the final timeline in which the changes are applied. This means it isn't the same order as the original and when sequences are not transactional this is causing issues. Some questions about if the way that the sequences are decoded is correct or not. There are maybe some bugs but if there are the question is if they can be fixed or if they are fundamental.<br />
<br />
Jeff - Is there consensus around what correct is in this context?<br />
<br />
Vondra - In principle, the same thing applies to logical replication in general- it should be the same as the commit order on the source database. For example, you shouldn't see a sequence going backwards. You shouldn't see an older version of the sequence.<br />
<br />
Jeff - Would you potentially see spurious updates that didn't make it to the primary? Would that be considered incorrect?<br />
<br />
Vondra - Don't see how you could get that and if it didn't make it into the transaction log then it didn't happen.<br />
<br />
Jeff - Perhaps we can discuss later<br />
<br />
Vondra - Happy to do that. Patch implementation has changed a few times. Eventually went back to the original simple decoding. Not sure how to improve that. Understand the concerns that have been raised on the list around the complexity of the interactions of these different timelines. Changes can happen in different orderings. In principle it might be wrong but that's very hard to prove. Not intent to just commit something to commit it and have to deal with bugs, want to discuss and try to solve them ahead of time.<br />
<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
<br />
Stephen - This continues to be worked on and there is recent activity on it.<br />
<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38656FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:32:30Z<p>Sfrost: /* Add the ability to limit the amount of memory that can be allocated to backends. */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
<br />
Stephen - This continues to be worked on and there is recent activity on it.<br />
<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38655FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:31:31Z<p>Sfrost: /* pg_rewind WAL deletion pitfall */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
<br />
Stephen - Last comment seems to indicate it isn't really being tested by CFbot? Maybe respond commenting on if it actually is or isn't.<br />
<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38654FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:28:40Z<p>Sfrost: /* ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
<br />
Drouvot - Will look at it again.<br />
<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38653FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:24:31Z<p>Sfrost: /* nbtree performance improvements through specialization on key shape */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
<br />
Matthias - What is remaining is a big patch to move functions around and then several patches which implement the specialization which are not very large in size. Apart from the first patch (240kb) which is code moving around, the patches are relatively small (23kb). Fairly straight-forward in terms of what it does. Only argument against it is that it adds binary size which might be a concern. Seems worth the effort because the performance is improved. Most interesting optimization have been split out into other patches which are more complex. Moves several functions into their own C files. Would like someone to review it in detail. Think it could maybe be done for v17 assuming there are not fundamental issues.<br />
<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38652FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:19:03Z<p>Sfrost: /* Add SPLIT PARTITION/MERGE PARTITIONS commands */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
<br />
Peter E - Looks like quite big patches.<br />
<br />
Nathan - Maybe need to suggest breaking into smaller patches, if possible.<br />
<br />
Peter E - Perhaps, though looks like may be a fairly complete patch<br />
<br />
Matthias - Was only for certain kinds of partitions? Don't think it covered RANGE partitions? Maybe because it is complex and hash partition splits are arguably easier<br />
<br />
Peter E - Maybe because what people want to do this for are hash partitions<br />
<br />
Vik - Looked at the changes and it didn't look that complicated on a very quick look<br />
<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38651FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:15:47Z<p>Sfrost: /* ALTER TABLE SET ACCESS METHOD on partitioned tables */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
<br />
Stephen - More in-depth commit message would be helpful - what does it do?<br />
<br />
Peter E - Probably can't specify access method on partitioned table today. Maybe this would set it so that it would then be inheirited, which seems like it's probably a reasonable thing.<br />
<br />
Vik - When applied to a partitioned table, new partitions will use this<br />
<br />
Peter E - Will take<br />
<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38650FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:13:29Z<p>Sfrost: /* Switching XLog source from archive to streaming when primary available */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
<br />
Nathan - Will look at it.<br />
<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38649FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:11:41Z<p>Sfrost: /* Add non-blocking version of PQcancel */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
<br />
Alvaro - Working on it<br />
<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38648FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:11:10Z<p>Sfrost: /* warn if GUC set to an invalid shared library */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
<br />
Berg - Added as reviewer<br />
<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38647FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:08:52Z<p>Sfrost: /* Speed up releasing of locks */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
<br />
Peter E - Folks involved can probably figure it out on their own<br />
<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38646FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:08:30Z<p>Sfrost: /* functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
<br />
Stephen - Seems like a pretty useful thing<br />
<br />
Peter E - Seems like there are a lot of details that need to be considered<br />
<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38645FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:05:30Z<p>Sfrost: /* AcquireExecutorLocks() and run-time pruning */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
<br />
Peter E - Basically waiting for review from Tom Lane<br />
<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38644FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:04:58Z<p>Sfrost: /* In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
<br />
Nathan - Some concerns around the 'undo log' addition<br />
<br />
Vondra - Was really just undo for one operation<br />
<br />
Nathan - Doesn't seem likely to be feasible for v17 though<br />
<br />
Vondra - A lot of other corner-cases with minimal WAL level<br />
<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38643FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:02:42Z<p>Sfrost: /* Parallelize correlated subqueries that execute within each worker */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
<br />
Peter E - Seems currently active.<br />
<br />
Vondra - Think this makes sense. It's possible to run these in each worker independently. Instead of disabling parallelism entirely.<br />
<br />
Munro - Bit like a nested loop join<br />
<br />
Vondra - Author is pretty good and if they need help then they will ask<br />
<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38642FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T15:00:49Z<p>Sfrost: /* Add foreign-server health checks infrastructure */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
<br />
Stephen - Seems like a useful thing<br />
<br />
Alvaro - Agreed<br />
<br />
Munro - This is based on the option where the server will check if the socket has gone away, so that the query could be killed early if the client has gone away. This does the same thing for checking foreign server connections. The way it was done for client connections is more portable than the way it's being done in this patch. This patch should probably adjust to how the client check is done so that it's more portable.<br />
<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38641FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T14:58:11Z<p>Sfrost: /* pg_stat_statements and "IN" conditions */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
<br />
Vondra - Desire is to normalize the values in the IN condition. Currently different number of IN parameters end up as different queries.<br />
<br />
Matthias - Concern is that it's a change in behavior, particularly with enum values.<br />
<br />
Alvaro - Just need a way to turn it off, which looks like there is an option for<br />
<br />
Peter E - Looks like it's pretty straight-forward<br />
<br />
Nathan - Earlier versions were trying to group by the the number of options which seemed very odd<br />
<br />
Vondra - Seems very arbitrary<br />
<br />
Alvaro - Maybe but could be useful<br />
<br />
Vondra - Maybe do the simple thing<br />
<br />
Nathan - Agreed on doing the simple thing and then maybe it can get in<br />
<br />
=== Add foreign-server health checks infrastructure ===<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38640FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T14:54:20Z<p>Sfrost: /* Function to log backtrace of postgres processes */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
<br />
Peter E - Signal handling implications, raises a concern<br />
<br />
Alvaro - Maybe you won't get the backtrace that you want in the case of a signal handler<br />
<br />
Vondra - Process gets a signal and then logs a backtrace during CFI<br />
<br />
Peter E - Not sure what this is useful for? In a production system you probably wouldn't end up getting much useful. In development, can just use a debugger.<br />
<br />
Jeff - Would have found it useful in some cases.<br />
<br />
Alvaro - Sometimes we have to ask users to get a backtrace and that is dangerous. This would be much safer.<br />
<br />
Vondra - If the CFI is very far away then it may not be very useful really.<br />
<br />
Peter E - Doesn't look like many committers have really commented on it. If someone wants to pick it up if someone is interested?<br />
<br />
(crickets)<br />
<br />
Bruce - Seems like it just may not be useful. Even if nothing is wrong with it we may not want it.<br />
<br />
Munro - Does seem like people are interested enough in it and perhaps it is useful.<br />
<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
=== Add foreign-server health checks infrastructure ===<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38639FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T14:48:49Z<p>Sfrost: /* session variables, LET command */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
=== Add foreign-server health checks infrastructure ===<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
<br />
Skip<br />
<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38638FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T14:48:27Z<p>Sfrost: /* More scalable multixacts buffers and locking */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
=== Add foreign-server health checks infrastructure ===<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
<br />
Alvaro- Will take this<br />
<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38637FOSDEM/PGDay 2024 Developer Meeting2024-02-01T14:47:32Z<p>Sfrost: /* v17 Patch Triage */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:30<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:30-13:40<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:40-13:50<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:white;"<br />
|Thur 13:50-14:00<br />
|Thomas' other thing.<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 15:00<br />
|Tea<br />
<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:30-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.<br />
<br />
Peter E - What is the difference between this and the C.UTF-8 locale?<br />
<br />
Jeff - The C.UTF-8 has changed the collation before, for one thing.<br />
<br />
Peter E - Example?<br />
<br />
Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.<br />
<br />
Peter E - More of a bootstrap problem?<br />
<br />
Munro - Yeah, may be able to just ignore that issue as just being a historical problem.<br />
<br />
Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.<br />
<br />
Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.<br />
<br />
Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.<br />
<br />
Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.<br />
<br />
Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.<br />
<br />
Peter E - Who is actually using that?<br />
<br />
Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.<br />
<br />
Matthias - If I don't care about natural ordering but just care about seeing similar things together.<br />
<br />
Peter E - Maybe as a DBA but probably not the case as an application developer.<br />
<br />
Matthias - I don't really care about real collation in my apps generally.<br />
<br />
Berg - If you're running a server for the world then perhaps it's fine.<br />
<br />
Peter E - Maybe we do this for v1 but in v2 add in full support.<br />
<br />
Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.<br />
<br />
Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.<br />
<br />
Munro - A good number of different things are just symlinks between each other.<br />
<br />
Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.<br />
<br />
Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.<br />
<br />
Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.<br />
<br />
Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.<br />
<br />
Matthias - Recently ICU had a release where they changed but they didn't change the identifier.<br />
<br />
Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.<br />
<br />
Munro - All those other ones suck though and they're poorly understood.<br />
<br />
Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.<br />
<br />
Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?<br />
<br />
Matthias - Using the GROUP BY with an ORDER BY might be much slower then<br />
<br />
Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.<br />
<br />
Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.<br />
<br />
Matthias - Original proposal was to make primary key text indexes use this<br />
<br />
Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.<br />
<br />
Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.<br />
<br />
Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.<br />
<br />
Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.<br />
<br />
Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.<br />
<br />
Jeff - Not quite type or what the user is going to do with it..<br />
<br />
Munro - User could say 'collate C'..<br />
<br />
Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.<br />
<br />
Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.<br />
<br />
Jeff - Database-level collation must be honored and so we would not be changing that.<br />
<br />
Vondra - What thought the problem was is that there was a misunderstanding around that.<br />
<br />
Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.<br />
<br />
Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.<br />
<br />
Peter E - Maybe use binary or something instead of C<br />
<br />
Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.<br />
<br />
Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.<br />
<br />
Peter E - I've now signed up as a reviewer...<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.<br />
<br />
Peter E - Proposed something similar before and therefore generally happy with it.<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
Nathan - MAINTAIN privilege originally slated for v16 but got reverted. Idea is to have it enable you to VACUUM, CLUSTER, LOCK (maybe others), and then a predefined role would exist that could allow that across all objects inside of a particular database. Reverted for v16 due to search_path trick concerns. Functional indexes could possibly end up causing issues. Patch is still applying cleanly. Big discussion is what to do about the search_path issue. A couple of options- reset the search_path to something 'safe' for all of the MAINTAIN sub-commands. This is already done for certain cases (like autovacuum). Downside of that is that the functionality might be changed sometimes. In practice, the autovacuum approach doesn't seem to have caused issues and maybe would be ok. Maybe only do this if you are not also the table owner though. Another option - maybe recommend that people set search_path within functions so that the function would always be run with that search_path. Issue is that that had performance issues, but work was done to try and improve that to deal with that performance problem. Last option is to just not do anything but maybe document this. Restricting search_path for maintenance commands seems simple and sufficient.<br />
<br />
Jeff - Setting search_path while executing maintenance commands is fine, but maybe have more explicit support and saying how vacuum/autovacuum is doing this. Still seems pretty weird- the reason autovacuum does this is a bit of a hack to deal with the security issue. What was done then to solve the security problem was sensible but it isn't really sensible overall and is kind of a hack. Our security model around this has evolved some and tried to deal with so many problems. Having trouble getting a good idea of what to do next. Alternative - what should we do about functions, what should the search path be, how should we run them. Maybe we have a search clause for a function which would define the origin of the search path- from the user invoking the function, or the system default, or the owner. Proposal didn't have a lot of traction though.<br />
<br />
Magnus - Issue is with expression indexes?<br />
<br />
Jeff - That's the most acute issue.<br />
<br />
Peter E - Functions in expression indexes which depend on the search_path ...? Seems like a terrible thing. Maybe just prohibit it.<br />
<br />
Jeff - Costly case is attaching search_path option to functions. MAINTAIN reverted because there might be a function that depends on the search path which would allow the table owner to gain access to the MAINTAIN user's account.<br />
<br />
Peter E - We could make the function fail if it tries to use the search_path. Issue was that it's expensive to set the search_path though?<br />
<br />
Jeff - Conclusion to make this safe was to set a search_path for their function but that was slow in v16. That was made a lot faster in v17 though and so that then becomes a more viable possible solution.<br />
<br />
Magnus - If no explicit search_path set on the function and the function used in an index then just set that to pg_catalog? Users could set their own search_path on the function if they want to.<br />
<br />
Munro - If there is not an explicit search_path set then maybe set it at CREATE FUNCTION time?<br />
<br />
Peter E - If we had done that originally but it's too late now perhaps.<br />
<br />
Stephen - Current situation ends up with things breaking when the search_path changes under a function anyway in some cases at run-time and so maybe we can just make this change.<br />
<br />
Munro - Maybe we could have it as a policy<br />
<br />
Magnus - Or have it be the default to be turned on (save search_path as part of CREATE FUNCTION), but then allow the option to turn that off perhaps as a GUC so it can be dealt with globally.<br />
<br />
Berg - Issue is that saving the entire search_path may not work because it could include other schemas which don't have an object there today but that object might be added later.<br />
<br />
Joe - Shadow issue exists due to conersion issues too not just explicit function in one schema and also in another, but could be inside of a schema. Column in table is varchar, you use lower() function, existing catalog function is lower(text), but then someone creates lower(varchar), the lower(varchar) will get used instead and that's still an issue.<br />
<br />
Stephen - End thought is at runtime when an expression index is used in some way, if the function in the index does not explicitly set a search_path then forcibly set it to pg_catalog.<br />
<br />
Berg - At CREATE INDEX time too<br />
<br />
Magnus - Yes<br />
<br />
Nathan - Seems about where it landed.<br />
<br />
Berg - Might break things in practice, but should be very clear<br />
<br />
Magnus - If it breaks things then it really needed to be fixed anyway.<br />
<br />
Jeff - Right now, if you do an INSERT and there's an index expression, the function will execute with the caller's search_path which is a problem.<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
Alvaro - We shouldn't move patches out of the CF just because it's ignored.<br />
<br />
Dave P - If there's no activity when assigned to author then boot them<br />
<br />
Alvaro - Sure that's fine, but we shouldn't boot out patches just because they haven't been responded to. Can be very de-motivating which is not helpful.<br />
<br />
Vik - Rule of 'review patch of equal size' when submitting patches should perhaps be enforced somehow. Maybe add something to the CF app to do it?<br />
<br />
Dave P - How would you practically enforce that?<br />
<br />
Peter E - Hard to keep up with even just updating the app with all of the state changes. Adding on analysis of who has reviewed what would make it even worse to keep it updated. Would take more time away from actually doing patch review.<br />
<br />
Vik - Some folks say applied patch and all tests run and count that as a review, which it is, but still needs more review.<br />
<br />
Dave C - Would like to review but not sure how to. Maybe something at PGConf.Dev will help with this?<br />
<br />
Matthias - There will be a workshop at PGConf.Dev about how to make patches more committable and hopefully also about reviewing patches.<br />
<br />
Dave C - A lot of unspoken rules ...<br />
<br />
Matthias - On the list with hundreds of messages a day ...<br />
<br />
Dave P - Trying to force people to do patch review will result in minimal reviews which wouldn't end up being helfpul.<br />
<br />
[many] Maybe not helpful to ask people to just download and apply the patch and run it these days because cfbot is already doing that. Would be helpful to have people check if there are tests for the new feature or the change, or if there is documentation ...<br />
<br />
Vondra - If the patch author does not explain what the patch does then few people are able to do a review. A junior person would struggle with the patch. Dealing with the reviewer bandwidth problem is to help reviewers with doing reviews and getting them to be proficient at it and this is part of the point of the workshop that's being done in Vancouver at PGConf.Dev. Want to explain the overall process and how it works and what tools are available for it. Only like an hour of talking and so not too deep into what you can do in reviews but to give an overall review of reviewing and what works and what doesn't. Only solution long-term is to increase the number of people who can do meaningful review.<br />
<br />
Dave P - Need to also set expectations and need to make sure people don't think that if they do a review then they'd definitely get a review themselves.<br />
<br />
Jeff - Can someone do something to get a patch closer to a point where a committer will be more likely to pick it up and so the committer would feel comfortable moving forward with the patch. Minor cleanup isn't likely to help very much, but an unresolved question on the mailing list would be helpful for a reviewer to highlight.<br />
<br />
Vik - There are meaningful partial reviews. Have reviewed patches on the SQL level, but don't look at the code, so signing off may not be ideal because the code also needs to be reviewed.<br />
<br />
Jeff - Committer looking at a SQL standard procedure would definitely be helped by a SQL person who has reviewed it and then checked the code.<br />
<br />
Joe - If goal is to get more people to do reviews - would it make sense to have a way to recognize official reviewers or in some official way?<br />
<br />
Berg - If the CF app had a way to take notes on a patch?<br />
<br />
Magnus - There is an annotate feature which will generate an email.<br />
<br />
Matthias - It isn't used very much.<br />
<br />
Berg - What is missing is a free text field?<br />
<br />
Stephen - Use the wiki?<br />
<br />
Dave P - Issue might be using mailing lists instead of other tools ...<br />
<br />
Berg - Perhaps a one-line summary as for what the actual state the patch is in<br />
<br />
Vondra - Perhaps for the summary if a specific email could be highlighted the indicates what the current state is<br />
<br />
Dave C - Who writes the summary though? A non-experienced reviewer likely wouldn't be able to write the summary.<br />
<br />
Vondra - The patch author should be the one providing the summary, or maybe someone experienced who really reviews the patch. Then provide a status of the patch. If things are left out then someone would need to point that out, of course.<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
Peter E - Probably should explain on the contributors page<br />
<br />
Vondra - Perhaps have a dedicated page for it that talks about the contributors committee, etc. Do not think that having rigid rules would make sense, informal is probably better, but we should explain in general to allow people to have some visibility.<br />
<br />
Dave P - There is a page on the wiki but it's pretty light-weight.<br />
<br />
Nathan - The committers page may also have some.<br />
<br />
Dave P - Contributors committee should propose a patch to pgweb to make it a proper policy.<br />
<br />
Vondra - Not too prescriptive but at least outline the process and explain for new people to have an understanding of how it works.<br />
<br />
Dave P - Sponsorship has a policy around this and might be a good model to use.<br />
<br />
=== Library dependency situation ===<br />
<br />
Munro - Didn't know enough about it at first when stepping up to work on it, but then I realized it was almost impossible that all of the versions would work because couldn't even get them all because there's so many different versions and tried to come up with a way to get rid of old versions so we don't have to continue to support them. Wanted to come up with a system to decide which versions to support. Daniel mentioned that this is a general problem and not specific to just LLVM. Don't have a way to decide which versions we actually support of various libraries. Having a machine in the buildfarm may be relevant but is kind of the tail wagging the dog and maybe we shouldn't just support what buildfarm supports- buildfarm should check what matters. Big Linux distributions have opinions on what should be supported but everything else doesn't really have an organization behind them to push specific support of things. Rules proposed for LLVM- take all the Linux distros in the buildfarm and take them as distributions that are interesting, then check the published EOL date for those operating systems. If the OS isn't supported anymore, then we don't care about it in terms of supporting it in newer versions of PG. Following through on that process with LLVM, and looking at possibly doing that for other libraries and dependencies. Just wanted to rant generally and raise the question about this and hopefully get to a point of having a policy where we can have little discussion about dropping things and instead just drop them.<br />
<br />
Dave P - Similar issues with pgAdmin and it has come down a lot to what version of python is being used. Have had to get very strict on it. Lots of difference between LLVM though and other things. RedHat will lock for a specific version of OpenSSL as an example and instead will back-port things. They do not consider LLVM to be the same and they will happily bump up the version of LLVM which has been a problem.<br />
<br />
Peter E - Is LLVM in the base system?<br />
<br />
Devrim - Yes, it is.<br />
<br />
Dave P - RPMs will be working towards fixing on a LLVM version.<br />
<br />
Devrim - Yes, work is ongoing for that.<br />
<br />
Munro - LLVM doesn't really have old versions, they just cut a new version every 6 months or so.<br />
<br />
Dave P - Sometimes upstream doesn't support older things in terms of libraries. Also with pl/python, the ecosystem will drop support for something like a crypto library and you won't be able to install the latest version because it's using python 3.6 or something.<br />
<br />
Peter E - Tried to put a chart together on the wiki as to what the requirements PG has vs. what OS's support what. Even figuring out what PG supports isn't easy. Going to continue to try to put together a wiki page of this which covers what version of PG supports what versions of what dependencies.<br />
<br />
Munro - Tried to build a database of this by scraping from the buildfarm and the OS vendor pages.<br />
<br />
Peter E - Not always guaranteed that will catch things, try to look into the RPM download directories<br />
<br />
Dave P - buildfarm animals sometimes pick up things like homebrew and end up with things different from what's on the OS.<br />
<br />
Munro - Seem like those can problably just be ignored generally or at least they don't seem to be as much of a concern.<br />
<br />
Dave P - Some of those things will just pull in the version that is supported whatever the latest is from the upstream. One of the issues with that though is that sometimes they do lack behind because a recipe hasn't been updated for a new thing.<br />
<br />
Munro - Things like homebrew are likely to have more-or-less current or new things.<br />
<br />
Dave C - buildfarm maybe isn't representative of what people are running<br />
<br />
Munro - People who don't have buildfarm animals for things they care about should probably add a new buildfarm animal.<br />
<br />
Dave P - Good point was made to not let the buildfarm drive what we support<br />
<br />
Munro - Yes, was able to remove a lot of code when we kicked some things out<br />
<br />
Peter E - Need to make sure that we talk about support as being the 'normal' one and not the super extended support<br />
<br />
Berg - Need to distinguish support for new major versions vs. back-branch versions of PG.<br />
<br />
Munro - If someone is using older versions and we do not want to break support for back-branches but that should all generally be just fine because those libraries won't be getting broken on those older systems either and we only are providing support for 5 years for our major versions.<br />
<br />
Jeff - Trying to figure out why we would need to support building new versions of PG on really old systems, don't think we really need that.<br />
<br />
Dave P - Right, that is basically what we are doing for pgAdmin, which has a list of OS's with what versions of pgAdmin are supported or expected to work on those OS versions.<br />
<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
[[FOSDEM/PGDay 2024 Developer Meeting Patch Review]]<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting_Patch_Review&diff=38636FOSDEM/PGDay 2024 Developer Meeting Patch Review2024-02-01T14:45:53Z<p>Sfrost: Created page with "== FOSDEM 2024 Developer Meeting Patch Review == == Bug Fixes == === Fixes for non-atomic read of read of control file on ext4 + ntfs === === Fix assertion failure in SnapBu..."</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting Patch Review ==<br />
<br />
== Bug Fixes ==<br />
<br />
=== Fixes for non-atomic read of read of control file on ext4 + ntfs ===<br />
=== Fix assertion failure in SnapBuildInitialSnapshot() ===<br />
=== Add some checks to avoid stack overflow ===<br />
=== Issue in postgres_fdw causing unnecessary wait for cancel request reply ===<br />
=== Postpone reparameterization of paths until when creating plans ===<br />
=== Archive current timeline history file on archiver startup if needed ===<br />
=== pg_ctl start may return 0 even if the postmaster has been already started on Windows ===<br />
=== [PATCH] LockAcquireExtended improvement ===<br />
=== Thread-safe gai_strerror() for Windows ===<br />
=== Forbid the use of invalidated physical slots in streaming replication. ===<br />
=== Fix COPY FROM...CSV importing \. on a line by itself ===<br />
=== "unexpected duplicate for tablespace" problem in logical replication ===<br />
=== Dump-restore loosing 'attnotnull' bit for DEFERRABLE PRIMARY KEY column(s) ===<br />
=== Network failure may prevent promotion ===<br />
=== False positive in bt_index_check in case of short 4B varlena datum ===<br />
=== Avoid deadlock and concurrency during orphan temp table removal ===<br />
<br />
== Clients ==<br />
<br />
=== functions to compute size of schemas/AMs (and maybe \dn++ and \dA++) ===<br />
=== Add non-blocking version of PQcancel ===<br />
=== Use libpq single row mode for FETCH_COUNT when a cursor cannot be used ===<br />
=== pgbench: allow to cancel queries during benchmark ===<br />
=== vacuumdb/clusterdb/reindexdb: allow specifying objects to process in all databases ===<br />
=== Improve tab completion for ALTER DEFAULT PRIVILEGE and ALTER TABLE ===<br />
=== Support for named parsed statement in psql ===<br />
=== pgbench log file headers ===<br />
=== Extend pgbench partitioning to pgbench_history ===<br />
=== psql: Rethinking of \du command ===<br />
=== Libpq Compression ===<br />
<br />
== Code Comments ==<br />
<br />
=== Fix comment about cross-checking the varnullingrels ===<br />
<br />
== Documentation ==<br />
<br />
=== Doc limitations update proposal: include out-of-line OID usage per TOAST-ed columns ===<br />
=== SET ROLE documentation improvement ===<br />
=== Various small unrelated doc patches: plpgsql, schemas, permissions, oidvector ===<br />
=== HOT - correct claim about indexes not referencing old line pointers ===<br />
=== Document that triggers can break foreign key constraints ===<br />
=== Add EXCLUDE COLLATE in CREATE/ALTER TABLE document ===<br />
=== Generate Doxygen documentation ===<br />
=== Add minimal C example and SQL registration example for custom table access methods. ===<br />
=== Add detail regarding resource consumption wrt max_connections ===<br />
=== Quick Start Guide to PL/pgSQL and PL/Python Documentation ===<br />
=== Simplify documentation related to Windows builds ===<br />
<br />
== Miscellaneous ==<br />
<br />
=== Function to log backtrace of postgres processes ===<br />
=== pgbench - adding pl/pgsql versions of tests ===<br />
=== archive modules loose ends ===<br />
=== pg_column_toast_chunk_id: a function to get a chunk ID of a TOASTed value ===<br />
=== Permute underscore separated components of columns before fuzzy matching ===<br />
=== Skip hidden files in serverside utilities ===<br />
=== Unlinking Parallel Hash Join inner batch files sooner ===<br />
=== Remove always true checks (src/backend/storage/buffer/bufmgr.c) ===<br />
=== Extension Enhancement: Buffer Invalidation in pg_buffercache ===<br />
=== Add pg_wait_for_lockers() function ===<br />
=== Add pg_get_owned_sequence function ===<br />
=== Reduce size of postgres.bki ===<br />
=== Improve the log message output of basic_archive when basic_archive.archive_directory parameter is not set ===<br />
=== Add pg_basetype() function to obtain a DOMAIN base type ===<br />
=== Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block ===<br />
=== make pg_ctl more friendly ===<br />
=== locked reads for atomics ===<br />
=== common signal handler protection ===<br />
=== Reducing serialized size of Nodes from nodeToString output ===<br />
=== Add SQL syntax check ===<br />
=== psql JSON output format ===<br />
=== Making the initial and maximum DSA segment sizes configurable ===<br />
=== Proposal to include --exclude-extension Flag in pg_dump ===<br />
=== Functions to return random numbers in a given range ===<br />
=== WIP Incremental Json Parser ===<br />
=== Tidy fill hstv array (src/backend/access/heap/pruneheap.c) ===<br />
=== Support a wildcard in backtrace_functions ===<br />
=== Add Index-level REINDEX with multiple jobs ===<br />
=== add function argument names to regex* functions. ===<br />
=== Add LSN <-> time conversion facility ===<br />
<br />
== Monitoring & Control ==<br />
<br />
=== pg_stat_statements and "IN" conditions ===<br />
=== Add foreign-server health checks infrastructure ===<br />
=== Amcheck verification of GiST and GIN ===<br />
=== Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE ===<br />
=== Add instrumentation showing if Heap2/PRUNE records came from VACUUM or from opportunistic pruning ===<br />
=== Logging parallel worker draught ===<br />
=== Logging plan of the currently running query ===<br />
=== Add last_commit_lsn to pg_stat_database ===<br />
=== Parent/child context relation in pg_get_backend_memory_contexts() ===<br />
=== Separate memory contexts for relcache and catcache ===<br />
=== Make it possible to add custom options to EXPLAIN ===<br />
=== Set log_lock_waits=on by default ===<br />
<br />
== Performance ==<br />
<br />
=== More scalable multixacts buffers and locking ===<br />
=== Parallelize correlated subqueries that execute within each worker ===<br />
=== In-place persistence change of a relation (fast ALTER TABLE ... SET LOGGED with wal_level=minimal) ===<br />
=== AcquireExecutorLocks() and run-time pruning ===<br />
=== Speed up releasing of locks ===<br />
=== nbtree performance improvements through specialization on key shape ===<br />
=== Add sortsupport for range types and btree_gist ===<br />
=== Improve dead tuple storage for lazy vacuum ===<br />
=== Reducing planning time when tables have many partitions ===<br />
=== ALTER TABLE and CLUSTER fail to use a BulkInsertState for toast tables ===<br />
=== Reduce timing overhead of EXPLAIN ANALYZE using rdtsc ===<br />
=== asynchronous execution support for Custom Scan ===<br />
=== BRIN Sort - sorting using BRIN indexes ===<br />
=== Check lateral references within PHVs for memoize cache keys ===<br />
=== pg_upgrade data type check connection overhead reduction ===<br />
=== Evaluate arguments of correlated SubPlans in the referencing ExprState ===<br />
=== Cross-database SERIALIZABLE safe snapshots ===<br />
=== A new strategy for pull-up correlated ANY_SUBLINK ===<br />
=== Support Right Semi Join ===<br />
=== Avoid unnecessary PlaceHolderVars for simple Vars ===<br />
=== XLog size reductions: smaller XLogRecordBlockHeader ===<br />
=== Index Prefetching ===<br />
=== Index-only filters without IOS ===<br />
=== Opportunistically pruning page before update ===<br />
=== XLog size reductions: Reduced XLog record header size ===<br />
=== Add bump memory context type and use it for tuplesorts ===<br />
=== Use ReadRecentBuffer() for btree root page ===<br />
=== Replace a large number of OR clauses with ANY expression ===<br />
=== Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans ===<br />
=== Extract numeric field in JSONB more effectively ===<br />
=== Memory consumed by child SpecialJoinInfo in partitionwise join planning ===<br />
=== Allow parallel plan for referential integrity checks ===<br />
=== Change tuple table slot for Unique node to "virtual" ===<br />
=== Streaming I/O, vectored I/O ===<br />
=== Statistics Import and Export ===<br />
=== Should consider materializing the cheapest inner path in consider_parallel_nestloop() ===<br />
=== Reuse child_relids in try_partitionwise_join ===<br />
=== Reducing memory consumed by RestrictInfo list translations in partitionwise join planning ===<br />
=== CRC32C Parallel Computation Optimization on ARM ===<br />
=== Index Insert Prefetching ===<br />
=== SLRU optimizations ===<br />
=== Special-case executor expression steps for common combinations ===<br />
=== nbtree: implement dynamic prefix truncation ===<br />
=== nbtree: downlink right separator/HIKEY optimization ===<br />
=== Properly pathify the UNION planner ===<br />
=== autovectorize page checksum code included elsewhere ===<br />
=== GUC hashtable optimizations ===<br />
=== Propagate pathkeys from CTEs up to the outer query ===<br />
=== add AVX2 support to simd.h ===<br />
=== Teach predtest about IS [NOT] <boolean> proofs ===<br />
=== 64-bit XIDs ===<br />
=== Adjust tuples estimate for appendrel ===<br />
=== Improve pg_dump/pg_restore/pg_upgrade handling of large objects ===<br />
=== Avoid computing ORDER BY junk columns unnecessarily ===<br />
=== An improvement on parallel DISTINCT ===<br />
=== Not to invalidate CatalogSnapshot for local invalidation messages ===<br />
=== Make vacuum opportunistic freezing adaptive ===<br />
=== shared detoast datum ===<br />
== Procedural Languages ==<br />
<br />
=== session variables, LET command ===<br />
=== bytea PL/Perl transform ===<br />
<br />
== Refactoring ==<br />
<br />
=== SetLatches() ===<br />
=== Use AF_UNIX for tests on Windows (ie drop fallback TCP code) ===<br />
=== Lock updated tuples in tuple_update() and tuple_delete() ===<br />
=== Revise get_cheapest_parallel_safe_total_inner ===<br />
=== Refactor pipe_read_line as a more generic interface for reading arbitrary strings off a pipe ===<br />
=== Use atomic ops for unlogged LSN ===<br />
=== Refactor fork+exec code ===<br />
=== Unified file access using virtual file descriptors ===<br />
=== Revises for the check of parameterized partial paths ===<br />
=== Retiring is_pushed_down ===<br />
=== Simplify create_merge_append_path a bit for clarity ===<br />
=== Move bki file pre-processing from initdb to bootstrap ===<br />
=== Relation bulk write facility ===<br />
=== New [relation] options engine ===<br />
=== automating RangeTblEntry node support ===<br />
=== Make attstattarget nullable ===<br />
=== change regexp_substr first argument make tests more easier to understand ===<br />
=== Confine vacuum skip logic to lazy_scan_skip ===<br />
== Replication & Recovery ==<br />
<br />
=== Switching XLog source from archive to streaming when primary available ===<br />
=== pg_rewind WAL deletion pitfall ===<br />
=== Improve WALRead() to suck data directly from WAL buffers when possible ===<br />
=== Make async slave to wait for lsn to be replayed ===<br />
=== Synchronizing slots from primary to standby ===<br />
=== Skip collecting decoded changes of already-aborted transactions ===<br />
=== Flush disk write caches by default on macOS and Windows ===<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
=== pg_rewind with cascade standby doesn't work well ===<br />
=== Force the old transactions logs cleanup even if checkpoint is skipped ===<br />
=== speed up a logical replication setup (pg_createsubscriber) ===<br />
=== Improve eviction algorithm in ReorderBuffer ===<br />
=== Flushing large data immediately in pqcomm ===<br />
<br />
== Security ==<br />
<br />
=== add not_before and not_after timestamps to sslinfo extension and pg_stat_ssl ===<br />
<br />
== Server Features ==<br />
<br />
=== ALTER TABLE SET ACCESS METHOD on partitioned tables ===<br />
=== BCP 47 locale names for Windows ===<br />
=== Provide the facility to set binary format output for specific OID's per session ===<br />
=== Patch to implement missing join selectivity estimation for range types ===<br />
=== logical decoding and replication of sequences, take 2 ===<br />
=== Add the ability to limit the amount of memory that can be allocated to backends. ===<br />
=== Multi-version ICU ===<br />
=== TDE key management patches ===<br />
=== Post-special Page Storage TDE support ===<br />
=== Transaction timeout ===<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
=== ltree hash functions ===<br />
=== UUID v7 ===<br />
=== Custom storage managers (SMGR), redux ===<br />
=== pg_stat_logmsg ===<br />
=== Forcing new tuples of updates off the page ===<br />
=== pg_tracing ===<br />
=== Support run-time partition pruning for hash join ===<br />
=== Support prepared statement invalidation when result or argument types change ===<br />
=== Allow INSTEAD OF DELETE triggers to modify the tuple for RETURNING ===<br />
=== Mark search_path as GUC_REPORT ===<br />
=== Add the page header to each SLRU page. ===<br />
=== Union Replacement of OR logic ===<br />
=== Add new protocol message to change GUCs to be able to change protocol extension parameters ===<br />
=== Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid ===<br />
=== Direct SSL connections ===<br />
=== Not to invaldiate CatalogSnapshot for local invalidation ===<br />
=== Remove useless GROUP BY columns considering unique index ===<br />
<br />
== SQL Commands ==<br />
<br />
=== Add SPLIT PARTITION/MERGE PARTITIONS commands ===<br />
=== MERGE INTO updatable_view ===<br />
=== MERGE ... WHEN NOT MATCHED BY SOURCE ===<br />
=== MERGE ... RETURNING ===<br />
=== Add CANONICAL option to xmlserialize ===<br />
=== Remaining sql/json patches ===<br />
=== Implement row pattern recognition feature ===<br />
=== COPY TO json ===<br />
=== RETURNING OLD/NEW values ===<br />
<br />
== System Administration ==<br />
<br />
=== warn if GUC set to an invalid shared library ===<br />
=== recovery modules ===<br />
=== Exponential backoff for auth_delay ===<br />
<br />
== Testing ==<br />
<br />
=== abidiff tests ===<br />
=== CI speed improvements for FreeBSD ===<br />
=== Build the docs if there are changes in docs and don't run other tasks if the changes are only in docs ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38635FOSDEM/PGDay 2024 Developer Meeting2024-02-01T13:18:19Z<p>Sfrost: </p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:30<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:30-13:40<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:40-13:50<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:white;"<br />
|Thur 13:50-14:00<br />
|Thomas' other thing.<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 15:00<br />
|Tea<br />
<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:30-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.<br />
<br />
Peter E - What is the difference between this and the C.UTF-8 locale?<br />
<br />
Jeff - The C.UTF-8 has changed the collation before, for one thing.<br />
<br />
Peter E - Example?<br />
<br />
Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.<br />
<br />
Peter E - More of a bootstrap problem?<br />
<br />
Munro - Yeah, may be able to just ignore that issue as just being a historical problem.<br />
<br />
Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.<br />
<br />
Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.<br />
<br />
Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.<br />
<br />
Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.<br />
<br />
Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.<br />
<br />
Peter E - Who is actually using that?<br />
<br />
Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.<br />
<br />
Matthias - If I don't care about natural ordering but just care about seeing similar things together.<br />
<br />
Peter E - Maybe as a DBA but probably not the case as an application developer.<br />
<br />
Matthias - I don't really care about real collation in my apps generally.<br />
<br />
Berg - If you're running a server for the world then perhaps it's fine.<br />
<br />
Peter E - Maybe we do this for v1 but in v2 add in full support.<br />
<br />
Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.<br />
<br />
Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.<br />
<br />
Munro - A good number of different things are just symlinks between each other.<br />
<br />
Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.<br />
<br />
Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.<br />
<br />
Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.<br />
<br />
Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.<br />
<br />
Matthias - Recently ICU had a release where they changed but they didn't change the identifier.<br />
<br />
Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.<br />
<br />
Munro - All those other ones suck though and they're poorly understood.<br />
<br />
Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.<br />
<br />
Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?<br />
<br />
Matthias - Using the GROUP BY with an ORDER BY might be much slower then<br />
<br />
Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.<br />
<br />
Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.<br />
<br />
Matthias - Original proposal was to make primary key text indexes use this<br />
<br />
Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.<br />
<br />
Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.<br />
<br />
Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.<br />
<br />
Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.<br />
<br />
Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.<br />
<br />
Jeff - Not quite type or what the user is going to do with it..<br />
<br />
Munro - User could say 'collate C'..<br />
<br />
Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.<br />
<br />
Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.<br />
<br />
Jeff - Database-level collation must be honored and so we would not be changing that.<br />
<br />
Vondra - What thought the problem was is that there was a misunderstanding around that.<br />
<br />
Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.<br />
<br />
Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.<br />
<br />
Peter E - Maybe use binary or something instead of C<br />
<br />
Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.<br />
<br />
Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.<br />
<br />
Peter E - I've now signed up as a reviewer...<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.<br />
<br />
Peter E - Proposed something similar before and therefore generally happy with it.<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
Nathan - MAINTAIN privilege originally slated for v16 but got reverted. Idea is to have it enable you to VACUUM, CLUSTER, LOCK (maybe others), and then a predefined role would exist that could allow that across all objects inside of a particular database. Reverted for v16 due to search_path trick concerns. Functional indexes could possibly end up causing issues. Patch is still applying cleanly. Big discussion is what to do about the search_path issue. A couple of options- reset the search_path to something 'safe' for all of the MAINTAIN sub-commands. This is already done for certain cases (like autovacuum). Downside of that is that the functionality might be changed sometimes. In practice, the autovacuum approach doesn't seem to have caused issues and maybe would be ok. Maybe only do this if you are not also the table owner though. Another option - maybe recommend that people set search_path within functions so that the function would always be run with that search_path. Issue is that that had performance issues, but work was done to try and improve that to deal with that performance problem. Last option is to just not do anything but maybe document this. Restricting search_path for maintenance commands seems simple and sufficient.<br />
<br />
Jeff - Setting search_path while executing maintenance commands is fine, but maybe have more explicit support and saying how vacuum/autovacuum is doing this. Still seems pretty weird- the reason autovacuum does this is a bit of a hack to deal with the security issue. What was done then to solve the security problem was sensible but it isn't really sensible overall and is kind of a hack. Our security model around this has evolved some and tried to deal with so many problems. Having trouble getting a good idea of what to do next. Alternative - what should we do about functions, what should the search path be, how should we run them. Maybe we have a search clause for a function which would define the origin of the search path- from the user invoking the function, or the system default, or the owner. Proposal didn't have a lot of traction though.<br />
<br />
Magnus - Issue is with expression indexes?<br />
<br />
Jeff - That's the most acute issue.<br />
<br />
Peter E - Functions in expression indexes which depend on the search_path ...? Seems like a terrible thing. Maybe just prohibit it.<br />
<br />
Jeff - Costly case is attaching search_path option to functions. MAINTAIN reverted because there might be a function that depends on the search path which would allow the table owner to gain access to the MAINTAIN user's account.<br />
<br />
Peter E - We could make the function fail if it tries to use the search_path. Issue was that it's expensive to set the search_path though?<br />
<br />
Jeff - Conclusion to make this safe was to set a search_path for their function but that was slow in v16. That was made a lot faster in v17 though and so that then becomes a more viable possible solution.<br />
<br />
Magnus - If no explicit search_path set on the function and the function used in an index then just set that to pg_catalog? Users could set their own search_path on the function if they want to.<br />
<br />
Munro - If there is not an explicit search_path set then maybe set it at CREATE FUNCTION time?<br />
<br />
Peter E - If we had done that originally but it's too late now perhaps.<br />
<br />
Stephen - Current situation ends up with things breaking when the search_path changes under a function anyway in some cases at run-time and so maybe we can just make this change.<br />
<br />
Munro - Maybe we could have it as a policy<br />
<br />
Magnus - Or have it be the default to be turned on (save search_path as part of CREATE FUNCTION), but then allow the option to turn that off perhaps as a GUC so it can be dealt with globally.<br />
<br />
Berg - Issue is that saving the entire search_path may not work because it could include other schemas which don't have an object there today but that object might be added later.<br />
<br />
Joe - Shadow issue exists due to conersion issues too not just explicit function in one schema and also in another, but could be inside of a schema. Column in table is varchar, you use lower() function, existing catalog function is lower(text), but then someone creates lower(varchar), the lower(varchar) will get used instead and that's still an issue.<br />
<br />
Stephen - End thought is at runtime when an expression index is used in some way, if the function in the index does not explicitly set a search_path then forcibly set it to pg_catalog.<br />
<br />
Berg - At CREATE INDEX time too<br />
<br />
Magnus - Yes<br />
<br />
Nathan - Seems about where it landed.<br />
<br />
Berg - Might break things in practice, but should be very clear<br />
<br />
Magnus - If it breaks things then it really needed to be fixed anyway.<br />
<br />
Jeff - Right now, if you do an INSERT and there's an index expression, the function will execute with the caller's search_path which is a problem.<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
Alvaro - We shouldn't move patches out of the CF just because it's ignored.<br />
<br />
Dave P - If there's no activity when assigned to author then boot them<br />
<br />
Alvaro - Sure that's fine, but we shouldn't boot out patches just because they haven't been responded to. Can be very de-motivating which is not helpful.<br />
<br />
Vik - Rule of 'review patch of equal size' when submitting patches should perhaps be enforced somehow. Maybe add something to the CF app to do it?<br />
<br />
Dave P - How would you practically enforce that?<br />
<br />
Peter E - Hard to keep up with even just updating the app with all of the state changes. Adding on analysis of who has reviewed what would make it even worse to keep it updated. Would take more time away from actually doing patch review.<br />
<br />
Vik - Some folks say applied patch and all tests run and count that as a review, which it is, but still needs more review.<br />
<br />
Dave C - Would like to review but not sure how to. Maybe something at PGConf.Dev will help with this?<br />
<br />
Matthias - There will be a workshop at PGConf.Dev about how to make patches more committable and hopefully also about reviewing patches.<br />
<br />
Dave C - A lot of unspoken rules ...<br />
<br />
Matthias - On the list with hundreds of messages a day ...<br />
<br />
Dave P - Trying to force people to do patch review will result in minimal reviews which wouldn't end up being helfpul.<br />
<br />
[many] Maybe not helpful to ask people to just download and apply the patch and run it these days because cfbot is already doing that. Would be helpful to have people check if there are tests for the new feature or the change, or if there is documentation ...<br />
<br />
Vondra - If the patch author does not explain what the patch does then few people are able to do a review. A junior person would struggle with the patch. Dealing with the reviewer bandwidth problem is to help reviewers with doing reviews and getting them to be proficient at it and this is part of the point of the workshop that's being done in Vancouver at PGConf.Dev. Want to explain the overall process and how it works and what tools are available for it. Only like an hour of talking and so not too deep into what you can do in reviews but to give an overall review of reviewing and what works and what doesn't. Only solution long-term is to increase the number of people who can do meaningful review.<br />
<br />
Dave P - Need to also set expectations and need to make sure people don't think that if they do a review then they'd definitely get a review themselves.<br />
<br />
Jeff - Can someone do something to get a patch closer to a point where a committer will be more likely to pick it up and so the committer would feel comfortable moving forward with the patch. Minor cleanup isn't likely to help very much, but an unresolved question on the mailing list would be helpful for a reviewer to highlight.<br />
<br />
Vik - There are meaningful partial reviews. Have reviewed patches on the SQL level, but don't look at the code, so signing off may not be ideal because the code also needs to be reviewed.<br />
<br />
Jeff - Committer looking at a SQL standard procedure would definitely be helped by a SQL person who has reviewed it and then checked the code.<br />
<br />
Joe - If goal is to get more people to do reviews - would it make sense to have a way to recognize official reviewers or in some official way?<br />
<br />
Berg - If the CF app had a way to take notes on a patch?<br />
<br />
Magnus - There is an annotate feature which will generate an email.<br />
<br />
Matthias - It isn't used very much.<br />
<br />
Berg - What is missing is a free text field?<br />
<br />
Stephen - Use the wiki?<br />
<br />
Dave P - Issue might be using mailing lists instead of other tools ...<br />
<br />
Berg - Perhaps a one-line summary as for what the actual state the patch is in<br />
<br />
Vondra - Perhaps for the summary if a specific email could be highlighted the indicates what the current state is<br />
<br />
Dave C - Who writes the summary though? A non-experienced reviewer likely wouldn't be able to write the summary.<br />
<br />
Vondra - The patch author should be the one providing the summary, or maybe someone experienced who really reviews the patch. Then provide a status of the patch. If things are left out then someone would need to point that out, of course.<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
Peter E - Probably should explain on the contributors page<br />
<br />
Vondra - Perhaps have a dedicated page for it that talks about the contributors committee, etc. Do not think that having rigid rules would make sense, informal is probably better, but we should explain in general to allow people to have some visibility.<br />
<br />
Dave P - There is a page on the wiki but it's pretty light-weight.<br />
<br />
Nathan - The committers page may also have some.<br />
<br />
Dave P - Contributors committee should propose a patch to pgweb to make it a proper policy.<br />
<br />
Vondra - Not too prescriptive but at least outline the process and explain for new people to have an understanding of how it works.<br />
<br />
Dave P - Sponsorship has a policy around this and might be a good model to use.<br />
<br />
=== Library dependency situation ===<br />
<br />
Munro - Didn't know enough about it at first when stepping up to work on it, but then I realized it was almost impossible that all of the versions would work because couldn't even get them all because there's so many different versions and tried to come up with a way to get rid of old versions so we don't have to continue to support them. Wanted to come up with a system to decide which versions to support. Daniel mentioned that this is a general problem and not specific to just LLVM. Don't have a way to decide which versions we actually support of various libraries. Having a machine in the buildfarm may be relevant but is kind of the tail wagging the dog and maybe we shouldn't just support what buildfarm supports- buildfarm should check what matters. Big Linux distributions have opinions on what should be supported but everything else doesn't really have an organization behind them to push specific support of things. Rules proposed for LLVM- take all the Linux distros in the buildfarm and take them as distributions that are interesting, then check the published EOL date for those operating systems. If the OS isn't supported anymore, then we don't care about it in terms of supporting it in newer versions of PG. Following through on that process with LLVM, and looking at possibly doing that for other libraries and dependencies. Just wanted to rant generally and raise the question about this and hopefully get to a point of having a policy where we can have little discussion about dropping things and instead just drop them.<br />
<br />
Dave P - Similar issues with pgAdmin and it has come down a lot to what version of python is being used. Have had to get very strict on it. Lots of difference between LLVM though and other things. RedHat will lock for a specific version of OpenSSL as an example and instead will back-port things. They do not consider LLVM to be the same and they will happily bump up the version of LLVM which has been a problem.<br />
<br />
Peter E - Is LLVM in the base system?<br />
<br />
Devrim - Yes, it is.<br />
<br />
Dave P - RPMs will be working towards fixing on a LLVM version.<br />
<br />
Devrim - Yes, work is ongoing for that.<br />
<br />
Munro - LLVM doesn't really have old versions, they just cut a new version every 6 months or so.<br />
<br />
Dave P - Sometimes upstream doesn't support older things in terms of libraries. Also with pl/python, the ecosystem will drop support for something like a crypto library and you won't be able to install the latest version because it's using python 3.6 or something.<br />
<br />
Peter E - Tried to put a chart together on the wiki as to what the requirements PG has vs. what OS's support what. Even figuring out what PG supports isn't easy. Going to continue to try to put together a wiki page of this which covers what version of PG supports what versions of what dependencies.<br />
<br />
Munro - Tried to build a database of this by scraping from the buildfarm and the OS vendor pages.<br />
<br />
Peter E - Not always guaranteed that will catch things, try to look into the RPM download directories<br />
<br />
Dave P - buildfarm animals sometimes pick up things like homebrew and end up with things different from what's on the OS.<br />
<br />
Munro - Seem like those can problably just be ignored generally or at least they don't seem to be as much of a concern.<br />
<br />
Dave P - Some of those things will just pull in the version that is supported whatever the latest is from the upstream. One of the issues with that though is that sometimes they do lack behind because a recipe hasn't been updated for a new thing.<br />
<br />
Munro - Things like homebrew are likely to have more-or-less current or new things.<br />
<br />
Dave C - buildfarm maybe isn't representative of what people are running<br />
<br />
Munro - People who don't have buildfarm animals for things they care about should probably add a new buildfarm animal.<br />
<br />
Dave P - Good point was made to not let the buildfarm drive what we support<br />
<br />
Munro - Yes, was able to remove a lot of code when we kicked some things out<br />
<br />
Peter E - Need to make sure that we talk about support as being the 'normal' one and not the super extended support<br />
<br />
Berg - Need to distinguish support for new major versions vs. back-branch versions of PG.<br />
<br />
Munro - If someone is using older versions and we do not want to break support for back-branches but that should all generally be just fine because those libraries won't be getting broken on those older systems either and we only are providing support for 5 years for our major versions.<br />
<br />
Jeff - Trying to figure out why we would need to support building new versions of PG on really old systems, don't think we really need that.<br />
<br />
Dave P - Right, that is basically what we are doing for pgAdmin, which has a list of OS's with what versions of pgAdmin are supported or expected to work on those OS versions.<br />
<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38633FOSDEM/PGDay 2024 Developer Meeting2024-02-01T13:00:50Z<p>Sfrost: /* Recognizing New Contributors */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:30<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:30-13:40<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:40-13:50<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:white;"<br />
|Thur 13:50-14:00<br />
|Thomas' other thing.<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 15:00<br />
|Tea<br />
<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:30-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.<br />
<br />
Peter E - What is the difference between this and the C.UTF-8 locale?<br />
<br />
Jeff - The C.UTF-8 has changed the collation before, for one thing.<br />
<br />
Peter E - Example?<br />
<br />
Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.<br />
<br />
Peter E - More of a bootstrap problem?<br />
<br />
Munro - Yeah, may be able to just ignore that issue as just being a historical problem.<br />
<br />
Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.<br />
<br />
Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.<br />
<br />
Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.<br />
<br />
Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.<br />
<br />
Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.<br />
<br />
Peter E - Who is actually using that?<br />
<br />
Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.<br />
<br />
Matthias - If I don't care about natural ordering but just care about seeing similar things together.<br />
<br />
Peter E - Maybe as a DBA but probably not the case as an application developer.<br />
<br />
Matthias - I don't really care about real collation in my apps generally.<br />
<br />
Berg - If you're running a server for the world then perhaps it's fine.<br />
<br />
Peter E - Maybe we do this for v1 but in v2 add in full support.<br />
<br />
Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.<br />
<br />
Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.<br />
<br />
Munro - A good number of different things are just symlinks between each other.<br />
<br />
Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.<br />
<br />
Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.<br />
<br />
Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.<br />
<br />
Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.<br />
<br />
Matthias - Recently ICU had a release where they changed but they didn't change the identifier.<br />
<br />
Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.<br />
<br />
Munro - All those other ones suck though and they're poorly understood.<br />
<br />
Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.<br />
<br />
Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?<br />
<br />
Matthias - Using the GROUP BY with an ORDER BY might be much slower then<br />
<br />
Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.<br />
<br />
Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.<br />
<br />
Matthias - Original proposal was to make primary key text indexes use this<br />
<br />
Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.<br />
<br />
Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.<br />
<br />
Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.<br />
<br />
Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.<br />
<br />
Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.<br />
<br />
Jeff - Not quite type or what the user is going to do with it..<br />
<br />
Munro - User could say 'collate C'..<br />
<br />
Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.<br />
<br />
Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.<br />
<br />
Jeff - Database-level collation must be honored and so we would not be changing that.<br />
<br />
Vondra - What thought the problem was is that there was a misunderstanding around that.<br />
<br />
Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.<br />
<br />
Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.<br />
<br />
Peter E - Maybe use binary or something instead of C<br />
<br />
Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.<br />
<br />
Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.<br />
<br />
Peter E - I've now signed up as a reviewer...<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.<br />
<br />
Peter E - Proposed something similar before and therefore generally happy with it.<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
Nathan - MAINTAIN privilege originally slated for v16 but got reverted. Idea is to have it enable you to VACUUM, CLUSTER, LOCK (maybe others), and then a predefined role would exist that could allow that across all objects inside of a particular database. Reverted for v16 due to search_path trick concerns. Functional indexes could possibly end up causing issues. Patch is still applying cleanly. Big discussion is what to do about the search_path issue. A couple of options- reset the search_path to something 'safe' for all of the MAINTAIN sub-commands. This is already done for certain cases (like autovacuum). Downside of that is that the functionality might be changed sometimes. In practice, the autovacuum approach doesn't seem to have caused issues and maybe would be ok. Maybe only do this if you are not also the table owner though. Another option - maybe recommend that people set search_path within functions so that the function would always be run with that search_path. Issue is that that had performance issues, but work was done to try and improve that to deal with that performance problem. Last option is to just not do anything but maybe document this. Restricting search_path for maintenance commands seems simple and sufficient.<br />
<br />
Jeff - Setting search_path while executing maintenance commands is fine, but maybe have more explicit support and saying how vacuum/autovacuum is doing this. Still seems pretty weird- the reason autovacuum does this is a bit of a hack to deal with the security issue. What was done then to solve the security problem was sensible but it isn't really sensible overall and is kind of a hack. Our security model around this has evolved some and tried to deal with so many problems. Having trouble getting a good idea of what to do next. Alternative - what should we do about functions, what should the search path be, how should we run them. Maybe we have a search clause for a function which would define the origin of the search path- from the user invoking the function, or the system default, or the owner. Proposal didn't have a lot of traction though.<br />
<br />
Magnus - Issue is with expression indexes?<br />
<br />
Jeff - That's the most acute issue.<br />
<br />
Peter E - Functions in expression indexes which depend on the search_path ...? Seems like a terrible thing. Maybe just prohibit it.<br />
<br />
Jeff - Costly case is attaching search_path option to functions. MAINTAIN reverted because there might be a function that depends on the search path which would allow the table owner to gain access to the MAINTAIN user's account.<br />
<br />
Peter E - We could make the function fail if it tries to use the search_path. Issue was that it's expensive to set the search_path though?<br />
<br />
Jeff - Conclusion to make this safe was to set a search_path for their function but that was slow in v16. That was made a lot faster in v17 though and so that then becomes a more viable possible solution.<br />
<br />
Magnus - If no explicit search_path set on the function and the function used in an index then just set that to pg_catalog? Users could set their own search_path on the function if they want to.<br />
<br />
Munro - If there is not an explicit search_path set then maybe set it at CREATE FUNCTION time?<br />
<br />
Peter E - If we had done that originally but it's too late now perhaps.<br />
<br />
Stephen - Current situation ends up with things breaking when the search_path changes under a function anyway in some cases at run-time and so maybe we can just make this change.<br />
<br />
Munro - Maybe we could have it as a policy<br />
<br />
Magnus - Or have it be the default to be turned on (save search_path as part of CREATE FUNCTION), but then allow the option to turn that off perhaps as a GUC so it can be dealt with globally.<br />
<br />
Berg - Issue is that saving the entire search_path may not work because it could include other schemas which don't have an object there today but that object might be added later.<br />
<br />
Joe - Shadow issue exists due to conersion issues too not just explicit function in one schema and also in another, but could be inside of a schema. Column in table is varchar, you use lower() function, existing catalog function is lower(text), but then someone creates lower(varchar), the lower(varchar) will get used instead and that's still an issue.<br />
<br />
Stephen - End thought is at runtime when an expression index is used in some way, if the function in the index does not explicitly set a search_path then forcibly set it to pg_catalog.<br />
<br />
Berg - At CREATE INDEX time too<br />
<br />
Magnus - Yes<br />
<br />
Nathan - Seems about where it landed.<br />
<br />
Berg - Might break things in practice, but should be very clear<br />
<br />
Magnus - If it breaks things then it really needed to be fixed anyway.<br />
<br />
Jeff - Right now, if you do an INSERT and there's an index expression, the function will execute with the caller's search_path which is a problem.<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
Alvaro - We shouldn't move patches out of the CF just because it's ignored.<br />
<br />
Dave P - If there's no activity when assigned to author then boot them<br />
<br />
Alvaro - Sure that's fine, but we shouldn't boot out patches just because they haven't been responded to. Can be very de-motivating which is not helpful.<br />
<br />
Vik - Rule of 'review patch of equal size' when submitting patches should perhaps be enforced somehow. Maybe add something to the CF app to do it?<br />
<br />
Dave P - How would you practically enforce that?<br />
<br />
Peter E - Hard to keep up with even just updating the app with all of the state changes. Adding on analysis of who has reviewed what would make it even worse to keep it updated. Would take more time away from actually doing patch review.<br />
<br />
Vik - Some folks say applied patch and all tests run and count that as a review, which it is, but still needs more review.<br />
<br />
Dave C - Would like to review but not sure how to. Maybe something at PGConf.Dev will help with this?<br />
<br />
Matthias - There will be a workshop at PGConf.Dev about how to make patches more committable and hopefully also about reviewing patches.<br />
<br />
Dave C - A lot of unspoken rules ...<br />
<br />
Matthias - On the list with hundreds of messages a day ...<br />
<br />
Dave P - Trying to force people to do patch review will result in minimal reviews which wouldn't end up being helfpul.<br />
<br />
[many] Maybe not helpful to ask people to just download and apply the patch and run it these days because cfbot is already doing that. Would be helpful to have people check if there are tests for the new feature or the change, or if there is documentation ...<br />
<br />
Vondra - If the patch author does not explain what the patch does then few people are able to do a review. A junior person would struggle with the patch. Dealing with the reviewer bandwidth problem is to help reviewers with doing reviews and getting them to be proficient at it and this is part of the point of the workshop that's being done in Vancouver at PGConf.Dev. Want to explain the overall process and how it works and what tools are available for it. Only like an hour of talking and so not too deep into what you can do in reviews but to give an overall review of reviewing and what works and what doesn't. Only solution long-term is to increase the number of people who can do meaningful review.<br />
<br />
Dave P - Need to also set expectations and need to make sure people don't think that if they do a review then they'd definitely get a review themselves.<br />
<br />
Jeff - Can someone do something to get a patch closer to a point where a committer will be more likely to pick it up and so the committer would feel comfortable moving forward with the patch. Minor cleanup isn't likely to help very much, but an unresolved question on the mailing list would be helpful for a reviewer to highlight.<br />
<br />
Vik - There are meaningful partial reviews. Have reviewed patches on the SQL level, but don't look at the code, so signing off may not be ideal because the code also needs to be reviewed.<br />
<br />
Jeff - Committer looking at a SQL standard procedure would definitely be helped by a SQL person who has reviewed it and then checked the code.<br />
<br />
Joe - If goal is to get more people to do reviews - would it make sense to have a way to recognize official reviewers or in some official way?<br />
<br />
Berg - If the CF app had a way to take notes on a patch?<br />
<br />
Magnus - There is an annotate feature which will generate an email.<br />
<br />
Matthias - It isn't used very much.<br />
<br />
Berg - What is missing is a free text field?<br />
<br />
Stephen - Use the wiki?<br />
<br />
Dave P - Issue might be using mailing lists instead of other tools ...<br />
<br />
Berg - Perhaps a one-line summary as for what the actual state the patch is in<br />
<br />
Vondra - Perhaps for the summary if a specific email could be highlighted the indicates what the current state is<br />
<br />
Dave C - Who writes the summary though? A non-experienced reviewer likely wouldn't be able to write the summary.<br />
<br />
Vondra - The patch author should be the one providing the summary, or maybe someone experienced who really reviews the patch. Then provide a status of the patch. If things are left out then someone would need to point that out, of course.<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
Peter E - Probably should explain on the contributors page<br />
<br />
Vondra - Perhaps have a dedicated page for it that talks about the contributors committee, etc. Do not think that having rigid rules would make sense, informal is probably better, but we should explain in general to allow people to have some visibility.<br />
<br />
Dave P - There is a page on the wiki but it's pretty light-weight.<br />
<br />
Nathan - The committers page may also have some.<br />
<br />
Dave P - Contributors committee should propose a patch to pgweb to make it a proper policy.<br />
<br />
Vondra - Not too prescriptive but at least outline the process and explain for new people to have an understanding of how it works.<br />
<br />
Dave P - Sponsorship has a policy around this and might be a good model to use.<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38631FOSDEM/PGDay 2024 Developer Meeting2024-02-01T12:53:11Z<p>Sfrost: /* Moving Forward with Pending Patches */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:30<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:30-13:40<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:40-13:50<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:white;"<br />
|Thur 13:50-14:00<br />
|Thomas' other thing.<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 15:00<br />
|Tea<br />
<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:30-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.<br />
<br />
Peter E - What is the difference between this and the C.UTF-8 locale?<br />
<br />
Jeff - The C.UTF-8 has changed the collation before, for one thing.<br />
<br />
Peter E - Example?<br />
<br />
Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.<br />
<br />
Peter E - More of a bootstrap problem?<br />
<br />
Munro - Yeah, may be able to just ignore that issue as just being a historical problem.<br />
<br />
Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.<br />
<br />
Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.<br />
<br />
Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.<br />
<br />
Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.<br />
<br />
Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.<br />
<br />
Peter E - Who is actually using that?<br />
<br />
Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.<br />
<br />
Matthias - If I don't care about natural ordering but just care about seeing similar things together.<br />
<br />
Peter E - Maybe as a DBA but probably not the case as an application developer.<br />
<br />
Matthias - I don't really care about real collation in my apps generally.<br />
<br />
Berg - If you're running a server for the world then perhaps it's fine.<br />
<br />
Peter E - Maybe we do this for v1 but in v2 add in full support.<br />
<br />
Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.<br />
<br />
Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.<br />
<br />
Munro - A good number of different things are just symlinks between each other.<br />
<br />
Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.<br />
<br />
Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.<br />
<br />
Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.<br />
<br />
Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.<br />
<br />
Matthias - Recently ICU had a release where they changed but they didn't change the identifier.<br />
<br />
Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.<br />
<br />
Munro - All those other ones suck though and they're poorly understood.<br />
<br />
Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.<br />
<br />
Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?<br />
<br />
Matthias - Using the GROUP BY with an ORDER BY might be much slower then<br />
<br />
Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.<br />
<br />
Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.<br />
<br />
Matthias - Original proposal was to make primary key text indexes use this<br />
<br />
Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.<br />
<br />
Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.<br />
<br />
Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.<br />
<br />
Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.<br />
<br />
Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.<br />
<br />
Jeff - Not quite type or what the user is going to do with it..<br />
<br />
Munro - User could say 'collate C'..<br />
<br />
Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.<br />
<br />
Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.<br />
<br />
Jeff - Database-level collation must be honored and so we would not be changing that.<br />
<br />
Vondra - What thought the problem was is that there was a misunderstanding around that.<br />
<br />
Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.<br />
<br />
Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.<br />
<br />
Peter E - Maybe use binary or something instead of C<br />
<br />
Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.<br />
<br />
Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.<br />
<br />
Peter E - I've now signed up as a reviewer...<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.<br />
<br />
Peter E - Proposed something similar before and therefore generally happy with it.<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
Nathan - MAINTAIN privilege originally slated for v16 but got reverted. Idea is to have it enable you to VACUUM, CLUSTER, LOCK (maybe others), and then a predefined role would exist that could allow that across all objects inside of a particular database. Reverted for v16 due to search_path trick concerns. Functional indexes could possibly end up causing issues. Patch is still applying cleanly. Big discussion is what to do about the search_path issue. A couple of options- reset the search_path to something 'safe' for all of the MAINTAIN sub-commands. This is already done for certain cases (like autovacuum). Downside of that is that the functionality might be changed sometimes. In practice, the autovacuum approach doesn't seem to have caused issues and maybe would be ok. Maybe only do this if you are not also the table owner though. Another option - maybe recommend that people set search_path within functions so that the function would always be run with that search_path. Issue is that that had performance issues, but work was done to try and improve that to deal with that performance problem. Last option is to just not do anything but maybe document this. Restricting search_path for maintenance commands seems simple and sufficient.<br />
<br />
Jeff - Setting search_path while executing maintenance commands is fine, but maybe have more explicit support and saying how vacuum/autovacuum is doing this. Still seems pretty weird- the reason autovacuum does this is a bit of a hack to deal with the security issue. What was done then to solve the security problem was sensible but it isn't really sensible overall and is kind of a hack. Our security model around this has evolved some and tried to deal with so many problems. Having trouble getting a good idea of what to do next. Alternative - what should we do about functions, what should the search path be, how should we run them. Maybe we have a search clause for a function which would define the origin of the search path- from the user invoking the function, or the system default, or the owner. Proposal didn't have a lot of traction though.<br />
<br />
Magnus - Issue is with expression indexes?<br />
<br />
Jeff - That's the most acute issue.<br />
<br />
Peter E - Functions in expression indexes which depend on the search_path ...? Seems like a terrible thing. Maybe just prohibit it.<br />
<br />
Jeff - Costly case is attaching search_path option to functions. MAINTAIN reverted because there might be a function that depends on the search path which would allow the table owner to gain access to the MAINTAIN user's account.<br />
<br />
Peter E - We could make the function fail if it tries to use the search_path. Issue was that it's expensive to set the search_path though?<br />
<br />
Jeff - Conclusion to make this safe was to set a search_path for their function but that was slow in v16. That was made a lot faster in v17 though and so that then becomes a more viable possible solution.<br />
<br />
Magnus - If no explicit search_path set on the function and the function used in an index then just set that to pg_catalog? Users could set their own search_path on the function if they want to.<br />
<br />
Munro - If there is not an explicit search_path set then maybe set it at CREATE FUNCTION time?<br />
<br />
Peter E - If we had done that originally but it's too late now perhaps.<br />
<br />
Stephen - Current situation ends up with things breaking when the search_path changes under a function anyway in some cases at run-time and so maybe we can just make this change.<br />
<br />
Munro - Maybe we could have it as a policy<br />
<br />
Magnus - Or have it be the default to be turned on (save search_path as part of CREATE FUNCTION), but then allow the option to turn that off perhaps as a GUC so it can be dealt with globally.<br />
<br />
Berg - Issue is that saving the entire search_path may not work because it could include other schemas which don't have an object there today but that object might be added later.<br />
<br />
Joe - Shadow issue exists due to conersion issues too not just explicit function in one schema and also in another, but could be inside of a schema. Column in table is varchar, you use lower() function, existing catalog function is lower(text), but then someone creates lower(varchar), the lower(varchar) will get used instead and that's still an issue.<br />
<br />
Stephen - End thought is at runtime when an expression index is used in some way, if the function in the index does not explicitly set a search_path then forcibly set it to pg_catalog.<br />
<br />
Berg - At CREATE INDEX time too<br />
<br />
Magnus - Yes<br />
<br />
Nathan - Seems about where it landed.<br />
<br />
Berg - Might break things in practice, but should be very clear<br />
<br />
Magnus - If it breaks things then it really needed to be fixed anyway.<br />
<br />
Jeff - Right now, if you do an INSERT and there's an index expression, the function will execute with the caller's search_path which is a problem.<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
Alvaro - We shouldn't move patches out of the CF just because it's ignored.<br />
<br />
Dave P - If there's no activity when assigned to author then boot them<br />
<br />
Alvaro - Sure that's fine, but we shouldn't boot out patches just because they haven't been responded to. Can be very de-motivating which is not helpful.<br />
<br />
Vik - Rule of 'review patch of equal size' when submitting patches should perhaps be enforced somehow. Maybe add something to the CF app to do it?<br />
<br />
Dave P - How would you practically enforce that?<br />
<br />
Peter E - Hard to keep up with even just updating the app with all of the state changes. Adding on analysis of who has reviewed what would make it even worse to keep it updated. Would take more time away from actually doing patch review.<br />
<br />
Vik - Some folks say applied patch and all tests run and count that as a review, which it is, but still needs more review.<br />
<br />
Dave C - Would like to review but not sure how to. Maybe something at PGConf.Dev will help with this?<br />
<br />
Matthias - There will be a workshop at PGConf.Dev about how to make patches more committable and hopefully also about reviewing patches.<br />
<br />
Dave C - A lot of unspoken rules ...<br />
<br />
Matthias - On the list with hundreds of messages a day ...<br />
<br />
Dave P - Trying to force people to do patch review will result in minimal reviews which wouldn't end up being helfpul.<br />
<br />
[many] Maybe not helpful to ask people to just download and apply the patch and run it these days because cfbot is already doing that. Would be helpful to have people check if there are tests for the new feature or the change, or if there is documentation ...<br />
<br />
Vondra - If the patch author does not explain what the patch does then few people are able to do a review. A junior person would struggle with the patch. Dealing with the reviewer bandwidth problem is to help reviewers with doing reviews and getting them to be proficient at it and this is part of the point of the workshop that's being done in Vancouver at PGConf.Dev. Want to explain the overall process and how it works and what tools are available for it. Only like an hour of talking and so not too deep into what you can do in reviews but to give an overall review of reviewing and what works and what doesn't. Only solution long-term is to increase the number of people who can do meaningful review.<br />
<br />
Dave P - Need to also set expectations and need to make sure people don't think that if they do a review then they'd definitely get a review themselves.<br />
<br />
Jeff - Can someone do something to get a patch closer to a point where a committer will be more likely to pick it up and so the committer would feel comfortable moving forward with the patch. Minor cleanup isn't likely to help very much, but an unresolved question on the mailing list would be helpful for a reviewer to highlight.<br />
<br />
Vik - There are meaningful partial reviews. Have reviewed patches on the SQL level, but don't look at the code, so signing off may not be ideal because the code also needs to be reviewed.<br />
<br />
Jeff - Committer looking at a SQL standard procedure would definitely be helped by a SQL person who has reviewed it and then checked the code.<br />
<br />
Joe - If goal is to get more people to do reviews - would it make sense to have a way to recognize official reviewers or in some official way?<br />
<br />
Berg - If the CF app had a way to take notes on a patch?<br />
<br />
Magnus - There is an annotate feature which will generate an email.<br />
<br />
Matthias - It isn't used very much.<br />
<br />
Berg - What is missing is a free text field?<br />
<br />
Stephen - Use the wiki?<br />
<br />
Dave P - Issue might be using mailing lists instead of other tools ...<br />
<br />
Berg - Perhaps a one-line summary as for what the actual state the patch is in<br />
<br />
Vondra - Perhaps for the summary if a specific email could be highlighted the indicates what the current state is<br />
<br />
Dave C - Who writes the summary though? A non-experienced reviewer likely wouldn't be able to write the summary.<br />
<br />
Vondra - The patch author should be the one providing the summary, or maybe someone experienced who really reviews the patch. Then provide a status of the patch. If things are left out then someone would need to point that out, of course.<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38630FOSDEM/PGDay 2024 Developer Meeting2024-02-01T11:32:14Z<p>Sfrost: /* The Path to un-reverting the MAINTAIN privilege */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:30<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:30-13:40<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:40-13:50<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:white;"<br />
|Thur 13:50-14:00<br />
|Thomas' other thing.<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 15:00<br />
|Tea<br />
<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:30-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.<br />
<br />
Peter E - What is the difference between this and the C.UTF-8 locale?<br />
<br />
Jeff - The C.UTF-8 has changed the collation before, for one thing.<br />
<br />
Peter E - Example?<br />
<br />
Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.<br />
<br />
Peter E - More of a bootstrap problem?<br />
<br />
Munro - Yeah, may be able to just ignore that issue as just being a historical problem.<br />
<br />
Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.<br />
<br />
Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.<br />
<br />
Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.<br />
<br />
Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.<br />
<br />
Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.<br />
<br />
Peter E - Who is actually using that?<br />
<br />
Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.<br />
<br />
Matthias - If I don't care about natural ordering but just care about seeing similar things together.<br />
<br />
Peter E - Maybe as a DBA but probably not the case as an application developer.<br />
<br />
Matthias - I don't really care about real collation in my apps generally.<br />
<br />
Berg - If you're running a server for the world then perhaps it's fine.<br />
<br />
Peter E - Maybe we do this for v1 but in v2 add in full support.<br />
<br />
Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.<br />
<br />
Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.<br />
<br />
Munro - A good number of different things are just symlinks between each other.<br />
<br />
Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.<br />
<br />
Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.<br />
<br />
Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.<br />
<br />
Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.<br />
<br />
Matthias - Recently ICU had a release where they changed but they didn't change the identifier.<br />
<br />
Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.<br />
<br />
Munro - All those other ones suck though and they're poorly understood.<br />
<br />
Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.<br />
<br />
Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?<br />
<br />
Matthias - Using the GROUP BY with an ORDER BY might be much slower then<br />
<br />
Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.<br />
<br />
Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.<br />
<br />
Matthias - Original proposal was to make primary key text indexes use this<br />
<br />
Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.<br />
<br />
Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.<br />
<br />
Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.<br />
<br />
Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.<br />
<br />
Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.<br />
<br />
Jeff - Not quite type or what the user is going to do with it..<br />
<br />
Munro - User could say 'collate C'..<br />
<br />
Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.<br />
<br />
Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.<br />
<br />
Jeff - Database-level collation must be honored and so we would not be changing that.<br />
<br />
Vondra - What thought the problem was is that there was a misunderstanding around that.<br />
<br />
Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.<br />
<br />
Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.<br />
<br />
Peter E - Maybe use binary or something instead of C<br />
<br />
Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.<br />
<br />
Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.<br />
<br />
Peter E - I've now signed up as a reviewer...<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.<br />
<br />
Peter E - Proposed something similar before and therefore generally happy with it.<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
Nathan - MAINTAIN privilege originally slated for v16 but got reverted. Idea is to have it enable you to VACUUM, CLUSTER, LOCK (maybe others), and then a predefined role would exist that could allow that across all objects inside of a particular database. Reverted for v16 due to search_path trick concerns. Functional indexes could possibly end up causing issues. Patch is still applying cleanly. Big discussion is what to do about the search_path issue. A couple of options- reset the search_path to something 'safe' for all of the MAINTAIN sub-commands. This is already done for certain cases (like autovacuum). Downside of that is that the functionality might be changed sometimes. In practice, the autovacuum approach doesn't seem to have caused issues and maybe would be ok. Maybe only do this if you are not also the table owner though. Another option - maybe recommend that people set search_path within functions so that the function would always be run with that search_path. Issue is that that had performance issues, but work was done to try and improve that to deal with that performance problem. Last option is to just not do anything but maybe document this. Restricting search_path for maintenance commands seems simple and sufficient.<br />
<br />
Jeff - Setting search_path while executing maintenance commands is fine, but maybe have more explicit support and saying how vacuum/autovacuum is doing this. Still seems pretty weird- the reason autovacuum does this is a bit of a hack to deal with the security issue. What was done then to solve the security problem was sensible but it isn't really sensible overall and is kind of a hack. Our security model around this has evolved some and tried to deal with so many problems. Having trouble getting a good idea of what to do next. Alternative - what should we do about functions, what should the search path be, how should we run them. Maybe we have a search clause for a function which would define the origin of the search path- from the user invoking the function, or the system default, or the owner. Proposal didn't have a lot of traction though.<br />
<br />
Magnus - Issue is with expression indexes?<br />
<br />
Jeff - That's the most acute issue.<br />
<br />
Peter E - Functions in expression indexes which depend on the search_path ...? Seems like a terrible thing. Maybe just prohibit it.<br />
<br />
Jeff - Costly case is attaching search_path option to functions. MAINTAIN reverted because there might be a function that depends on the search path which would allow the table owner to gain access to the MAINTAIN user's account.<br />
<br />
Peter E - We could make the function fail if it tries to use the search_path. Issue was that it's expensive to set the search_path though?<br />
<br />
Jeff - Conclusion to make this safe was to set a search_path for their function but that was slow in v16. That was made a lot faster in v17 though and so that then becomes a more viable possible solution.<br />
<br />
Magnus - If no explicit search_path set on the function and the function used in an index then just set that to pg_catalog? Users could set their own search_path on the function if they want to.<br />
<br />
Munro - If there is not an explicit search_path set then maybe set it at CREATE FUNCTION time?<br />
<br />
Peter E - If we had done that originally but it's too late now perhaps.<br />
<br />
Stephen - Current situation ends up with things breaking when the search_path changes under a function anyway in some cases at run-time and so maybe we can just make this change.<br />
<br />
Munro - Maybe we could have it as a policy<br />
<br />
Magnus - Or have it be the default to be turned on (save search_path as part of CREATE FUNCTION), but then allow the option to turn that off perhaps as a GUC so it can be dealt with globally.<br />
<br />
Berg - Issue is that saving the entire search_path may not work because it could include other schemas which don't have an object there today but that object might be added later.<br />
<br />
Joe - Shadow issue exists due to conersion issues too not just explicit function in one schema and also in another, but could be inside of a schema. Column in table is varchar, you use lower() function, existing catalog function is lower(text), but then someone creates lower(varchar), the lower(varchar) will get used instead and that's still an issue.<br />
<br />
Stephen - End thought is at runtime when an expression index is used in some way, if the function in the index does not explicitly set a search_path then forcibly set it to pg_catalog.<br />
<br />
Berg - At CREATE INDEX time too<br />
<br />
Magnus - Yes<br />
<br />
Nathan - Seems about where it landed.<br />
<br />
Berg - Might break things in practice, but should be very clear<br />
<br />
Magnus - If it breaks things then it really needed to be fixed anyway.<br />
<br />
Jeff - Right now, if you do an INSERT and there's an index expression, the function will execute with the caller's search_path which is a problem.<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38626FOSDEM/PGDay 2024 Developer Meeting2024-02-01T10:59:24Z<p>Sfrost: /* CREATE SUBSCRIPTION ... SERVER */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.<br />
<br />
Peter E - What is the difference between this and the C.UTF-8 locale?<br />
<br />
Jeff - The C.UTF-8 has changed the collation before, for one thing.<br />
<br />
Peter E - Example?<br />
<br />
Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.<br />
<br />
Peter E - More of a bootstrap problem?<br />
<br />
Munro - Yeah, may be able to just ignore that issue as just being a historical problem.<br />
<br />
Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.<br />
<br />
Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.<br />
<br />
Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.<br />
<br />
Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.<br />
<br />
Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.<br />
<br />
Peter E - Who is actually using that?<br />
<br />
Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.<br />
<br />
Matthias - If I don't care about natural ordering but just care about seeing similar things together.<br />
<br />
Peter E - Maybe as a DBA but probably not the case as an application developer.<br />
<br />
Matthias - I don't really care about real collation in my apps generally.<br />
<br />
Berg - If you're running a server for the world then perhaps it's fine.<br />
<br />
Peter E - Maybe we do this for v1 but in v2 add in full support.<br />
<br />
Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.<br />
<br />
Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.<br />
<br />
Munro - A good number of different things are just symlinks between each other.<br />
<br />
Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.<br />
<br />
Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.<br />
<br />
Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.<br />
<br />
Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.<br />
<br />
Matthias - Recently ICU had a release where they changed but they didn't change the identifier.<br />
<br />
Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.<br />
<br />
Munro - All those other ones suck though and they're poorly understood.<br />
<br />
Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.<br />
<br />
Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?<br />
<br />
Matthias - Using the GROUP BY with an ORDER BY might be much slower then<br />
<br />
Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.<br />
<br />
Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.<br />
<br />
Matthias - Original proposal was to make primary key text indexes use this<br />
<br />
Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.<br />
<br />
Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.<br />
<br />
Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.<br />
<br />
Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.<br />
<br />
Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.<br />
<br />
Jeff - Not quite type or what the user is going to do with it..<br />
<br />
Munro - User could say 'collate C'..<br />
<br />
Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.<br />
<br />
Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.<br />
<br />
Jeff - Database-level collation must be honored and so we would not be changing that.<br />
<br />
Vondra - What thought the problem was is that there was a misunderstanding around that.<br />
<br />
Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.<br />
<br />
Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.<br />
<br />
Peter E - Maybe use binary or something instead of C<br />
<br />
Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.<br />
<br />
Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.<br />
<br />
Peter E - I've now signed up as a reviewer...<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
Jeff - This is already being discussed on the list and it is changing a bit in direction thanks to that discussion. No real dispute around this patch. In the spirit of this meeting, decided to throw this topic out there in case someone wanted to discuss it. Not very controversial. Some of the original patch got some feedback about going in a bit of a different direction with only minor core changes and that's actually the direction that it's going in.<br />
<br />
Peter E - Proposed something similar before and therefore generally happy with it.<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38625FOSDEM/PGDay 2024 Developer Meeting2024-02-01T10:56:37Z<p>Sfrost: /* Built-in collation provider for "C" and "C.UTF-8" */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
Jeff - Talk about C.UTF-8 built-in collation provider. If you take aside collation and consider just other unicode functions- initcap, regular expressions, other parts of the backend use ascii sometimes and use the database default collation provider and use those for various things, adjusting characters to try to treat international characters better. All these other behaviors are still quite important and we are doing everything through libc or ICU and we can't document or test any of that in much the same way we do it for collations. We have all these problems not just for collations but also for all these behaviors because of the collation providers. This is difficult because you can't provide simple examples for functions. The other behavior is pretty easy to build in though. Collation carries a lot of other complications but these other unicode operations are generally just lookup tables with a few exceptions. Unless you're trying to localize upper/lower functions, in general it's basically always the same. There were very few cases where upper/lower did something different for every locale on a given system (eg: Turkish). Essentially saying that we could probably build-in these basic/default unicode behavior in a maintainable way by basically importing these lookup tables. We would also get a performance benefit by doing this. All of this setting aside collation, which does carry a lot of other complexity, but we could at least solve these non-collation behavior issues ourselves and solve that problem. We would still provide the ability to use libc or ICU for localized collation and other unicode behaviors but this would allow us to have our own built-in provider instead. This would be a better version of what C.UTF-8 locale offers, essentially, as it would be built-in and we could document it and test it. Have discussed it on the list and gotten some support for it from a useability standpoint and developer standpoint. Have also had some concerns raised but think those have been mainly responded to. Not a solution to the collation problem but instead all of the other issues around there. For collation, this would essentially be the C collation. For a database it's often difficult to chose a locale anyway. Often isn't really possible to pick a single collation for a whole database anyway.<br />
<br />
Peter E - What is the difference between this and the C.UTF-8 locale?<br />
<br />
Jeff - The C.UTF-8 has changed the collation before, for one thing.<br />
<br />
Peter E - Example?<br />
<br />
Munro - Independent libraries may have created C.UTF-8 locales and some project (Debian?) created one first and then glibc added support for it later but it was a bit different and then Debian dropped their patches for it and that changed things.<br />
<br />
Peter E - More of a bootstrap problem?<br />
<br />
Munro - Yeah, may be able to just ignore that issue as just being a historical problem.<br />
<br />
Jeff - Anyone who is using C.UTF-8 today, this built-in provider would just be flat-out better always. Wouldn't have the risk of such a historical thing happening, at least, and we would be matching the version of unicode that this locale uses with the version of unicode that PG uses for normalization. Risk of these changes would be low. As new versions of unicode come out, this would be updated. This would allow us to avoid those changes happening at the OS level under us and instead we could document it as part of PG releases. We could then also document these changes as part of PG documentation. Unicode has quite strong guarantees around this behavior but won't say that it won't ever break especially around undefined unicode codepoints.<br />
<br />
Munro - Think it's a really cool idea but for another reason- kill Windows locale support. It's completely unmaintained and it's unloved and would be great to just get rid of it and instead offer a built-in consistent option. If you don't want that, then you should just use ICU.<br />
<br />
Jeff - Other aspects - this would also be available on all platforms, such as Mac which doesn't have C.UTF-8. Available beyond just glibc users. Collation is still an important topic and you would probably still want to use ICU for collation as it's just preferable for natural language and it's also a lot better than libc. You could use COLLATE clauses to get that natural ordering that you want. Another benefit to that is that applying the COLLATE clause to the query itself avoids issues with indexes. The sort step would end up being the final thing. Quite often that could end up being a better performing plan, even if using ICU which is faster than libc, but it's still going to be slower than using C locale and if that is what matters for a particular operation that could be much better performing.<br />
<br />
Peter E - Hard disagree on that, can see the technical appeal of adding it just to have it but not sure that anyone would really actually use it. Expecting people to have to explicitly say COLLATE to get natural language sorting isn't going to go over well because we've been trying to get to a point where they don't have to explicitly ask for it.<br />
<br />
Jeff - This is more for people who are using C.UTF-8 now really. Not trying to say we're really changing any defaults at all. For people who have a database default collation of C.UTF-8 today this would make more sense.<br />
<br />
Peter E - Who is actually using that?<br />
<br />
Jeff - Don't have specific numbers but think it's a pretty normal configuration as it performs better.<br />
<br />
Matthias - If I don't care about natural ordering but just care about seeing similar things together.<br />
<br />
Peter E - Maybe as a DBA but probably not the case as an application developer.<br />
<br />
Matthias - I don't really care about real collation in my apps generally.<br />
<br />
Berg - If you're running a server for the world then perhaps it's fine.<br />
<br />
Peter E - Maybe we do this for v1 but in v2 add in full support.<br />
<br />
Jeff - Don't want to rule out that possibility. Reasonable thing to consider but the issue is then maintenance of it. Would need enough people comfortable with that part of the code base to be able to maintain it. The root collation - unicode has all sorts of defaults for everything and it calls them defaults and as an example: France region of the French language has the same collation as the English language of the English language in the US. Not a linguist but the collation order is the same in both cases and is just the 'root' collation. Unicode has these defaults and they're meant as a guide but they're careful about calling things a default but they do have some kind of a default. The root collation order would be a great natural language sort order to provide by default by the project, but ICU offers that. libc does not offer that. No way to get root collation order from libc.<br />
<br />
Peter E - No obvious way to ask for it but you could get it from libc by asking for like French.<br />
<br />
Munro - A good number of different things are just symlinks between each other.<br />
<br />
Jeff - Some people might just not include those locales. If there was to be a 'default default' then that might be a reasonable thing to have like ICU or we could consider what that would look like to build into PG overall. Happy to contribute and work on that and think it would be useful to go down that road, but don't want to promise that. Proposal for v17 is not that and don't want to promise that we would get there.<br />
<br />
Munro - The obvious alternative would be to just say use ICU more and perhaps make it the default too. We could provide a different provider that uses ICU code for things but then we would be using ICU's version.<br />
<br />
Jeff - One big benefit would be putting the unicode versions in lock-step which we wouldn't be able to do if we're using external dependencies. Big thing I like about this proposal is being able to document all of this, including things like being able to document how to do regexps with different locales. The idea that this would be documentable is a huge benefit but we can't get that with any dependency.<br />
<br />
Munro - Sounds great when you talk about just basic C type but when we start talking about taking this further then maybe we should just buy into ICU more.<br />
<br />
Matthias - Recently ICU had a release where they changed but they didn't change the identifier.<br />
<br />
Joe - Every other database has the option to have a built-in collation and locale support. PG is really the only one that doesn't.<br />
<br />
Munro - All those other ones suck though and they're poorly understood.<br />
<br />
Jeff - Concern raised about having a built-in root collation because that might blur the line between using ICU and using built-in provider. Agree with that and after thinking about it we are not likely to want the tailoring and localization as ICU is going to be better at that and would want to push people towards ICU. Trying to own all the issues with ICU seems like a lot of work.<br />
<br />
Berg - What you might do is have an internal techincal collation when the database does sorting on its own when it isn't asked for it (like for GROUP BY), maybe always use C locale for that?<br />
<br />
Matthias - Using the GROUP BY with an ORDER BY might be much slower then<br />
<br />
Stephen - But explicit idea is to only do this when not being asked for an ordering and no ORDER BY included.<br />
<br />
Jeff - Similar case for indexes, if it's only for internal usage and not for other cases, but there was concern about teaching the planner how to do this and choose the right option. You'd have to decorate paths with knowledge of where we can do this and where we can't do this and we would add complexity and possibly bugs into the planner around this. This might be able to be overcome though and could provide some serious performance benefits. Another way to think about this is that we do something similar for hashing operations- we only use the specific functions if the collation is non-deterministic but for deterministic collations we just use generic hashing.<br />
<br />
Matthias - Original proposal was to make primary key text indexes use this<br />
<br />
Jeff - Wasn't exactly part of the proposal but the thread took off a bit. Without bringing up the thread specifically and such, there was a realization that we are accepting a lot of potential downsides with the risks associated with using a non-C collation for indexes and in other cases because when a collation changes then the index breaks. Also imposing a performance cost associated with building the index and also the cost of the index becoming corrupted due to changes. What I was trying to get across in the thread was to think about this cost/benefit question between these risks and costs which are imposed on all of these text indexes which may result in only very small benefits.<br />
<br />
Joe - ICU in some cases it's 10x faster than glibc and so there's also the issue that many many people are using glibc.<br />
<br />
Bruce - Yeah, you're bringing in the cost in terms of performance and the corruption risk for pretty limited benefit.<br />
<br />
Jeff - Mostly for primary keys it's just an equality lookup and so this ends up being very impactful. There was a lot in the thread and we could also discuss later.<br />
<br />
Munro - Attempting to predict what the user is going to do with the data. Maybe have a distinction between text and 'human text' or such.<br />
<br />
Jeff - Not quite type or what the user is going to do with it..<br />
<br />
Munro - User could say 'collate C'..<br />
<br />
Jeff - Then you have a lot of extra typing to say COLLATE C for a lot of keys and then FKs, etc. All somewhat related at least. Very messy problem. ICU built by default in v16 which is a big step as libc is just really not good. More ICU would be great. One of the problems is that ICU doesn't support the C locale. Today we can't do away with libc support because a lot of people use C.UTF-8. With this proposal, people could use ICU or use the built-in provider and get rid of libc support.<br />
<br />
Vondra - One of the main problems that I had with the proposal is that it seemed like "if user does not specify locale, then if the user didn't specify the locale then we just change it to whatever is faster" which seems a bit strange. Reasonable expectation of user is that the database is created with a specific locale, then everything will use that as the default. A lot of users specify it at the database level and expect it to work using that locale for everything in that database. Would be ok with changing the implementation detail as long as it keeps the same result. Would be really surprising for users though would be changing to something faster but changes the ordering.<br />
<br />
Jeff - Database-level collation must be honored and so we would not be changing that.<br />
<br />
Vondra - What thought the problem was is that there was a misunderstanding around that.<br />
<br />
Jeff - Agreed that the discussion needs to be framed better to make it clear that this wasn't intended as impacting users in that way.<br />
<br />
Matthias - Regarding new built-in locale - not sure we should use the C or C.UTF-8 as the name.<br />
<br />
Peter E - Maybe use binary or something instead of C<br />
<br />
Matthias - It'll exist everywhere and naming it as C or C.UTF-8 is not very user friendly.<br />
<br />
Jeff - This capability isn't really accounted for in the SQL. Would be interesting to think about another way to specify the behavior of upper/lower. Could imagine one day maybe say just using the unicode data files and not have any dependency for upper/lower but also have support for things like the greek difference. Regarding renaming, pretty much fine with whatever name we pick. The SQL specifies only one example of the german capital S ... If we do choose names then we should try to leave room for possible variants.<br />
<br />
Peter E - I've now signed up as a reviewer...<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38624FOSDEM/PGDay 2024 Developer Meeting2024-02-01T10:04:23Z<p>Sfrost: /* Meson and packaging */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|Property Graph Queries (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Property Graph Queries ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
Devrim - Quick summary - started building packages 20-ish years ago, just the PG RPMS. Now we have haproxy, consul, lots of other things. 200-300 packages for updates now. Not about just meson but about most of the things. Want to create a connection between the packagers and the developers. Discussed ICU things with Jeff Davis in the past and with Munro regarding IBM things on the lists, this is great. When there is a new feature like llvm in PG11, having the hackers tell the packagers is very helpful and if the packagers don't know then the packagers may not include support for those new features. Packagers are reachable via wiki, packagers mailing list, etc. Regarding meson- not following every email but trying to follow those regarding meson. Wonder if PG17 will be built with meson? Not just about meson but packagers should be included in discussion regarding versions of libraries like zlib. We have a unified spec file for PG. First question- is meson good enough to be used for the PG17 package? What can we do to improve connection between hackers, users, and packagers. Not just about PG but about also includes postgis and the various libraries that it uses and other extensions too.<br />
<br />
Peter E - meson is not 100% functional with the make build system for non-Windows platforms. As of today, not sensible to use for official packages. Possible that these issues will be fixed in time for PG17 but then we would be switching at the last minute. Also need to check if all of the extensions work properly with meson build. Maybe for v18 we switch early in the cycle for it.<br />
<br />
Berg - Tried meson but noticed that it doesn't support llvm which is a giant feature and stopped because of that.<br />
<br />
Alvaro - Need to make sure that once meson stuff is installed and you want to install an extension and is much harder to ensure that the extension can be built like pgxs, does it have support for meson and work?<br />
<br />
Peter E - Maybe switch early in v18 cycle and try to fix it up.<br />
<br />
Berg - As long as it is supposed to work then it's about fixing bugs.<br />
<br />
Peter E - Not worthwhile since llvm support just doesn't work at all right now.<br />
<br />
Alvaro - Can we insist on disallowing cross-compilation?<br />
<br />
Munro - Packagers mailing list is currently restricted but maybe we need one that isn't restricted for packaging discussions?<br />
<br />
Peter E - People can send to packagers ... and maybe CC other lists<br />
<br />
Magnus - Gets weird CC'ing between lists that are private and not private<br />
<br />
Devrim - Not just about Berg and Devrim but there's lots of other packagers out there who are not just using the community packages. More people need to be aware of this. Experience so far is that not everyone is aware and we release and then other people come and ask Devrim about the new things in the spec file. Don't expect me to follow everything though.<br />
<br />
Peter E - Existing packaging lists have very little traffic currently. Packagers generally pull info from upstream through release notes and things, not the case that all of the hackers out there reach out to the packagers to tell them about new things.<br />
<br />
Devrim - If zlib is added as a specific library required then maybe the patch author should check out if the library is actually available on all of the different platforms or not.<br />
<br />
Berg - Maybe postgres can do better regarding having hackers communicating with our packagers. Should packagers generally be expected to add new features? Should some things not be enabled by default? What about checksums?<br />
<br />
Peter E - Probably should just turn on all build-time options, but run-time options should be left as the PG defaults.<br />
<br />
Bruce - Release notes are written for a broad audience. Not designed to show absolutely every build change or every config change.<br />
<br />
Munro - Perhaps there needs to be a new document.<br />
<br />
Bruce - Maybe. Sometimes things are not added, because something that most users aren't going to see doesn't need to be included and we want to have the document be reasonable in terms of size.<br />
<br />
Peter E - We do have a section of incompatible changes, maybe we should have that be better organized.<br />
<br />
Bruce - Probably would be best as a separate document.<br />
<br />
Matthias - release notes for incompatible changes is much more user-focused. Maybe put something like this at the end of the page.<br />
<br />
Bruce - How many people read the release notes? Stephen - Lots. Bruce - Only like dozens of people are needing this specific information while the release notes are for a much broader audience and so it would probably be better as a separate document.<br />
<br />
Jeff - Is the biggest issue the dependencies?<br />
<br />
Peter E - There are subsections not interesting to most people.. Bruce - Maybe we should remove those sections too then.<br />
<br />
Matthias - Minor release notes have things that are very detailed.<br />
<br />
Bruce - We include more detailed info because we don't expect them to re-test between minor versions, so we want to list things that people might run into when they do a minor release update.<br />
<br />
Matthias - Maybe we add a separate page for search-level compatibility issues, developer-level issues, extension authors. Extensions being impacted by changes to internal structures.<br />
<br />
Peter E - Would never finish the release notes to include all those.<br />
<br />
Stephen - Extension authors should be following hackers or committers if they're using internal structures.<br />
<br />
Devrim - We build the beta packages and so it would be good to have the info about new switches that are added into configure before then.<br />
<br />
Magnus - Do we have a process for adding things to buildfarm animals in terms of switches?<br />
<br />
Matthias - Hackers who add the feature tend to add that option to their buildfarm animals.<br />
<br />
Munro - Or the hacker adding things has to reach out to buildfarm animal maintainers and keep on them to fix their animals and add support.<br />
<br />
Matthias - When is there going to be a good guideline on how to build extensions with meson?<br />
<br />
Peter E - In the fullness of time.<br />
<br />
Matthias - Have that for make but don't see that for meson.<br />
<br />
Peter E - No need to actually do that really.<br />
<br />
Berg - Quite a few extensions are pretty small and some are using cmake and larger extensions need more research anyway and many extensions are too small to really benefit from meson anyway.<br />
<br />
Peter E - What would be interesting would be to get these extensions working with meson to get Windows support for them as many could work on Windows but don't just because of the build issues that meson might fix.<br />
<br />
Magnus - Would be good to have something like pgxs that's built on meson but pgxs is pretty fundamentally built on make files.<br />
<br />
Matthias - Tried to get make files working on mac, then tried to use meson for an extension, didn't get it to work, copied one of the files from the PG project and was able to make it work but wasn't ideal.<br />
<br />
Magnus - Would be nice to have a tool to take pgxs makefile and convert it to meson if possible. Simple makefiles it might be possible to do the conversion. More complex makefiles need additional research. Would be useful to have that tool though.<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38621FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:59:18Z<p>Sfrost: /* Binary format output per session */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
Dave C - Have some research to do.<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38620FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:58:46Z<p>Sfrost: /* Binary format output per session */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with. Maybe survey how many types have binary support and how many have text and how many have both. Does JDBC driver use simple queries sometime?<br />
<br />
Dave C - Yes, sometimes it does use simple queries.<br />
<br />
Alvaro - Why use simple query?<br />
<br />
Dave C - Extended query can do odd things sometimes. Mainly use it for isvalid (checking if the connection is valid).<br />
<br />
Peter E - Maybe change the client to always use extended query and then send every query saying to have the query be returned in binary.<br />
<br />
Dave C - Wasn't aware that it was possible to do everything as binary, very curious how that works.<br />
<br />
Matthias - How does the driver know about the new type? Register it?<br />
<br />
Peter E - Trying to get away from having to register it.<br />
<br />
Dave C - Shouldn't matter because you wouldn't have a query asking for something that the driver doesn't have a decoder for. Very few types that are actually new, most use floats, et al.<br />
<br />
Matthias - At least now you only use the binary types if you know you can handle it.<br />
<br />
Dave C - The person who wrote the application is going to know that they'll be able to handle the type that they are querying for.<br />
<br />
Magnus - Might have to make it optional in the driver if you're using an unknown extension.<br />
<br />
Dave C - Yeah, if something new was introduced then the app would break and have to be fixed if something new happened. Or it would use text.<br />
<br />
Alvaro - New developer adds new table with query and the app breaks or gets much slower?<br />
<br />
Dave C - Wouldn't get slower, developer would have to either add the decoder or actually select that it's text and then deal with it being slower. With Java/JDBC, application could register a new decoder.<br />
<br />
Munro - What about OIDs being reused? No validation machinery.<br />
<br />
Peter E - Nothing in the protocol handles that today. Maybe drivers already basically hard-code things.<br />
<br />
Matthias - For OIDs under the FirstValidUserOid (sp?) those won't ever change, only an issue for extensions adding/dropping extensions really.<br />
<br />
Discussion about having some kind of permanent identifier for types. (Peter E, Munro, Magnus, Matthias)<br />
<br />
Munro - Maybe some kind of number assigning authority..<br />
<br />
Matthias - Redo manager has a wiki page for extensions to use.<br />
<br />
Magnus - Works because there's very few users who need those.<br />
<br />
Peter E - All of this would require some protocol changes. Would still need to wait for the older things out there to die off that doesn't support protocol changes.<br />
<br />
Dave C - How old are those things?<br />
<br />
Joe - Maybe client that ships with RHEL 7 or something?<br />
<br />
Matthias - Any plans or ideas regarding update of our row send format because it's 4 bytes per field which is a lot of overhead but maybe we could reduce that somehow? No specific proposal for specific changes but am curious if there's interest in adding a new row message type for smaller data where the message itself has less overhead. Everything today works by handling up to a gigabyte per field but in some cases we don't need anywhere near that much. Stephen - Like VARLEN, Magnus - yea, Matthias - Similar to this, maybe also be able to do column compression somehow.<br />
<br />
Dave C - Have thought about larger protocol changes<br />
<br />
Peter E - From Robert - Flat out not viable to bump the protocol version (email from 2024-01-16).<br />
<br />
Dave C - v16 is the first version that really handles protocol changes?<br />
<br />
Peter E - Yeah but even then not sure that it really works. Certainly nothing before v16 in libpq would handle it. Other drivers would have to deal with it too.<br />
<br />
Dave C - Maybe can just say to use the latest version for JDBC, at least. This would mean we're locked into this for quite a while.<br />
<br />
Peter E - Much of this was back-patched in the middle of the prod releases and maybe if there's additional changes needed we could argue to back-patch those changes.<br />
<br />
Magnus - libpq needs to not fail with this is the main thing.<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38619FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:40:45Z<p>Sfrost: /* Binary format output per session */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
Dave C - Few years ago put out what all interfaces want. What all interfaces want is to get all info back in binary. Today everything comes back as text by default. JDBC and other drivers have to use extended query protocol and send a describe message (extra round-trip) to get data back in binary. In Java world, nearly everything comes back as text to avoid the extra round-trip. PGVector would benefit a lot from returning the data as binary instead of text due to it being lots of floats. Put together naive patch to use a GUC to pick OIDs to return as binary at the start of the session. Other interfaces have also looked at it- JDBC, npgsql (?) seem to like it. Doesn't seem like GUC is necessarily the best way to do this but would be good to do something.<br />
<br />
Peter E - Wrote email in Oct about this, quick summary: using a GUC is complicated because it ends up being best-effort. Many patches for this exist and need to consider connection poolers too. Discussion of making it a new protocol setting but we are not really ready to extend the protocol in the way we want to.<br />
<br />
Dave C - Don't see being able to extend the protocol?<br />
<br />
Peter E - Not sure. How to identify the types- names? OIDs? What is the OID ends up being different in different systems due to extensions? Haven't considered that part of it. Independent install of postgis into separate systems you'll get different OIDs, names you have to deal with schemas too, etc. One idea- Fixed UUIDv4 type that would be global? Want to have this be some kind of session property though. Don't want to ship this request with every query. Not a lot of session state in the protocol, not much precedent for it. Column-encryption has a similar issue. Have to hook session state in with the discard command and other things. Maybe could work similar to client encoding? Client communicates to server what client encoding it supports. Client encoding isn't terribly robust and there are concerns with using that approach but there are a lot of edge cases.<br />
<br />
Dave C - That's a challenge but the benefit is universal and quite large.<br />
<br />
Peter E - Could just make this a connection option.<br />
<br />
Dave C - That would be ok too.<br />
<br />
Matthias - No type that doesn't have binary.<br />
<br />
Peter E - Most do but some extensions may not.<br />
<br />
Matthias - When we flush the datum it's binary<br />
<br />
Peter E - No, it's the send/recv functions that we're talking about, not the internal.<br />
<br />
Dave C - Current default is we send everything as text.<br />
<br />
Nathan - If everything including extensions seems to have binary support then maybe that isn't a huge issue..<br />
<br />
Peter E - Maybe just have a flag that says everything comes back as binary?<br />
<br />
Dave C - Would work for me.<br />
<br />
Alvaro - What about pgbouncer?<br />
<br />
Peter E - Would still have to track it but would have only one flag to track.<br />
<br />
Magnus - Maybe have to have separate pools, one for binary one for text.<br />
<br />
Peter E - Need to think about it but maybe, but would probably be simpler with just one flag to deal with.<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38618FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:31:03Z<p>Sfrost: /* What's new in SQL:2023 */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the PG parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38617FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:26:35Z<p>Sfrost: /* What's new in SQL:2023 */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Tea<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
Discussion about implementation in the parser, how the definition of the property graphs works, what a PG implementation may look like, etc.<br />
<br />
=== Binary format output per session ===<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38615FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:20:37Z<p>Sfrost: /* What's new in SQL:2023 */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Coffee<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to turn GQL queries into SQL/PGQ.<br />
<br />
=== Binary format output per session ===<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38614FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:16:43Z<p>Sfrost: /* What's new in SQL:2023 */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Coffee<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on SQL/PGQ - Property Graph Queries which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model, talks about GQL (Graph Query Language), then about how to define graphs in SQL using SQL/PGQ, and how to query it with SQL.<br />
<br />
=== Binary format output per session ===<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38613FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:08:18Z<p>Sfrost: /* What's new in SQL:2023 */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Coffee<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
Vik's presentation on PGQ - Property Graph Query which is a new addition to SQL added into SQL 2023.<br />
<br />
Goes over differences between the relational model and the graph model.<br />
<br />
=== Binary format output per session ===<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38612FOSDEM/PGDay 2024 Developer Meeting2024-02-01T08:07:05Z<p>Sfrost: </p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 9:00-9:10<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:10-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Coffee<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
=== What's new in SQL:2023 ===<br />
<br />
=== Binary format output per session ===<br />
<br />
=== Meson and packaging ===<br />
<br />
=== Built-in collation provider for "C" and "C.UTF-8" ===<br />
<br />
=== CREATE SUBSCRIPTION ... SERVER ===<br />
<br />
=== The Path to un-reverting the MAINTAIN privilege ===<br />
<br />
=== Moving Forward with Pending Patches ===<br />
<br />
(Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
=== Recognizing New Contributors ===<br />
<br />
(Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
=== Page Features / Reserve space on page ===<br />
<br />
=== TDE / IVs / More ===<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
=== Any other business & close ===<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38610FOSDEM/PGDay 2024 Developer Meeting2024-01-31T16:29:47Z<p>Sfrost: /* FOSDEM 2024 Developer Meeting schedule by Time and Room */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Studio 4<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 8:30-9:00<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:00-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Coffee<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38609FOSDEM/PGDay 2024 Developer Meeting2024-01-31T16:26:22Z<p>Sfrost: </p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Madrid<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 8:30-9:30<br />
|Welcome and Introductions<br />
<br />
|-<br />
|Thur 9:00-9:30<br />
|What's new in SQL:2023 (Vik)<br />
<br />
|-<br />
|Thur 9:30-10:00<br />
|Binary format output per session (Dave C)<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|-<br />
|Thur 10:30-11:00<br />
|Meson and packaging (Devrim)<br />
<br />
|-<br />
|Thur 11:00-11:30<br />
|Built-in collation provider for "C" and "C.UTF-8" (Jeff)<br />
<br />
|-<br />
|Thur 11:30-12:00<br />
|CREATE SUBSCRIPTION ... SERVER (Jeff)<br />
<br />
|-<br />
|Thur 12:00-12:30<br />
|The Path to un-reverting the MAINTAIN privilege (Nathan)<br />
<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|-<br />
|Thur 13:00-13:30<br />
|Moving Forward with Pending Patches (Unknown: "I have encountered a situation where some of my patches have not been reviewed by others, preventing them from moving forward. I believe this is a common challenge faced by other developers as well. It would be great if we could engage in a discussion about this and potentially brainstorm ideas to improve it.").<br />
<br />
Note from Daniel: "This was proposed by a hacker who was unable to attend".<br />
<br />
|-<br />
|Thur 13:30-14:00<br />
|Recognizing New Contributors (Unknown: "Many active developers, including myself, desire to be listed as a Contributor, but just do not know how. This lack of clarity can be confusing. I'm wondering if it's possible to have a discussion on how to effectively recognize and acknowledge new contributors."). <br />
<br />
Note from Daniel: "This was proposed by that same person, and since he isn't able to attend I don't think it's fair to identify him (I did promise I'd bring this up anonymously to put focus on the issue and not the person). This is a person who I can vouch for being very active, prolific enough to show up on Roberts recent "Who contributed to.." blogpost."<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Coffee<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:00-14:30<br />
|Page Features / Reserve space on page<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 14:30-15:00<br />
|TDE / IVs / More<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 15:00-16:45<br />
|v17 Patch Triage<br />
|<br />
<br />
|- style="background-color:white;"<br />
|Thur 16:45-17:00<br />
|Any other business & close.<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=FOSDEM/PGDay_2024_Developer_Meeting&diff=38584FOSDEM/PGDay 2024 Developer Meeting2024-01-24T16:13:18Z<p>Sfrost: /* Suggested topics */</p>
<hr />
<div>== FOSDEM 2024 Developer Meeting schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Madrid<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 8:30-9:00<br />
|Welcome and Introductions<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 10:00<br />
|Coffee<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 12:30-13:00<br />
|Lunch<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00<br />
|Coffee<br />
<br />
|- style="background-color:lightgray;"<br />
|Thur 14:00-17:00<br />
|v17 Patch Triage<br />
|}<br />
<br />
== Suggested topics ==<br />
<br />
* Patch Triage<br />
* Moving Forward with Pending Patches<br />
* Recognizing New Contributors<br />
* What's new in SQL:2023<br />
* Binary format output per session<br />
* Meson and packaging<br />
* Developer Meeting during PGConf Europe<br />
* Page Features / Reserve space on page<br />
* TDE<br />
<br />
== Notes/Minutes ==<br />
<br />
<br />
=== v17 Patch Triage ===<br />
<br />
<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=PgCon_2023_Developer_Unconference&diff=37932PgCon 2023 Developer Unconference2023-06-02T15:00:57Z<p>Sfrost: </p>
<hr />
<div>This is the PGCon 2023 Unconf Page<br />
<br />
== Unconference schedule by Time and Room ==<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!DMS 1160<br />
!DMS 1120<br />
!DMS 1140<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 10:00-11:00<br />
|Choice of topics and votes<br />
|<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 11:15-12:15<br />
|Logical Replication<br />
|pg_hint_plan<br />
|Kernel Hacker<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 12:15-13:15<br />
|Lunch<br />
|<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 13:15-14:15<br />
|pg_upgrade<br />
|Build System<br />
|Cluster Awareness<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 14:15-14:30<br />
|Coffee Break<br />
|<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 14:30-15:30<br />
|Distributed Transactions<br />
|Vectors<br />
|Index Prefetching<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 15:30-15:45<br />
|Break<br />
|<br />
|<br />
<br />
|- style="background-color:lightgray;"<br />
|Fri 15:45-16:45<br />
|Future of PGCon<br />
|Table AMs<br />
|Partitioning<br />
|}<br />
<br />
== Notes/Minutes ==<br />
<br />
=== Slot 1 ===<br />
<br />
=== Slot 2 ===</div>Sfrosthttps://wiki.postgresql.org/index.php?title=PgCon_2023_Developer_Meeting&diff=37915PgCon 2023 Developer Meeting2023-05-30T16:02:05Z<p>Sfrost: </p>
<hr />
<div>A meeting of the interested PostgreSQL developers is being planned for Tuesday 30 May, 2023 at the University of Ottawa, prior to pgCon 2023. In order to keep the numbers manageable, this meeting is by '''invitation only'''.<br />
Any questions regarding the invitations to this event should be directed to the team of individuals tasked with coming up with the list of people to invite:<br />
<br />
* Andres Freund<br />
* Stephen Frost<br />
* Dave Page<br />
<br />
An Unconference will be held on Friday for in-depth discussion of technical topics.<br />
<br />
This is a PostgreSQL Community event.<br />
<br />
== Meeting Goals ==<br />
<br />
* Define the schedule for the upcoming releases<br />
* Address any proposed timing, policy, or procedure issues<br />
* Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.<br />
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br />
<br />
== Time & Location ==<br />
<br />
The meeting will (probably) be:<br />
<br />
* 9:00AM to 12PM<br />
* DMS 3105 - Desmarais Hall, 55 Laurier Avenue East<br />
* University of Ottawa.<br />
<br />
Lunch will be served during the meeting.<br />
<br />
== COVID-19 ==<br />
<br />
The University of Ottawa's COVID-19 guidance can be found at https://www.uottawa.ca/en/covid-19. Wearing of masks at the Developer Meeting will be optional, however we do ask that people do not attend if they have COVID symptoms or have tested positive.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!<br />
<br />
# Nathan Bossart<br />
# Joe Conway<br />
# Jeff Davis<br />
# Mark Dilger<br />
# Peter Eisentraut<br />
# Andres Freund<br />
# Stephen Frost<br />
# Etsuro Fujita<br />
# Peter Geoghegan<br />
# Magnus Hagander<br />
# Amit Kapila<br />
# Jonathan Katz<br />
# Alexander Kukushkin<br />
# Tom Lane<br />
# Heikki Linnakangas<br />
# Noah Misch<br />
# Thomas Munro<br />
# Dave Page<br />
# Michael Paquier<br />
# Melanie Plageman<br />
# Masahiko Sawada<br />
# Peter Smith<br />
# Tomas Vondra<br />
<br />
The following people will not be in Ottawa, and do not plan to attend:<br />
<br />
# Masao Fujii<br />
# Daniel Gustafsson<br />
# Álvaro Herrera<br />
# Tatsuo Ishii<br />
# Alexander Korotkov<br />
# Amit Langote<br />
# Dean Rasheed<br />
# David Rowley<br />
<br />
== Agenda Items ==<br />
''Please add suggestions for agenda items here. (with your name)''<br />
<br />
* 16.0 release and commitfest schedule (Dave)<br />
* Renaming "master" branch to "main"? (Michael)<br />
* A brief PG15 RMT postmortem and what can we improve? (Jonathan)<br />
* What are the big challenges for our users? What are the big challenges for us to solve? (Jonathan)<br />
* Cloud operators have data of value to this community, e.g. frequency of errors that should never happen. What sort of data sharing regime might provide a good balance between value to the community and comfort for cloud operators? (Noah)<br />
* High level thoughts and feedback on moving toward ICU as the preferred collation provider (Jeff)<br />
<br />
==Agenda==<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Item<br />
!Presenter<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|09:00 - 09:10<br />
|Welcome and introductions<br />
|Dave Page<br />
<br />
|- <br />
|09:10 - 09:20<br />
|Release and commitfest schedules<br />
|Dave Page<br />
<br />
|- <br />
|09:20 - 09:35<br />
|Renaming "master" branch to "main"<br />
|Michael Paquier<br />
<br />
|- <br />
|09:35 - 10:00<br />
|A brief PG15 RMT postmortem and what can we improve?<br />
|Jonathan Katz<br />
<br />
|- <br />
|10:00 - 10:30<br />
|What are the big challenges for our users? What are the big challenges for us to solve?<br />
|Jonathan Katz<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|10:30 - 11:00<br />
|Coffee break<br />
|All<br />
<br />
|- <br />
|11:00 - 11:20<br />
|High level thoughts and feedback on moving toward ICU as the preferred collation provider [[StateOfICU]]<br />
|Jeff Davis<br />
<br />
|- <br />
|11:20 - 11:50<br />
|Cloud operators have data of value to this community, e.g. frequency of errors that should never happen. What sort of data sharing regime might provide a good balance between value to the community and comfort for cloud operators? <br />
|Noah Misch<br />
<br />
|- <br />
|11:50 - 12:00<br />
|Any other business<br />
|Dave Page<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|12:00<br />
|Lunch<br />
|<br />
<br />
|}<br />
<br />
Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move materially however). Any remaining time before lunch may be used for Commitfest item triage or other activities.<br />
<br />
== Minutes ==<br />
Stephen Frost recording<br />
<br />
=== Welcome and Introductions ===<br />
Dave Page - General opening<br />
Introductions<br />
<br />
=== Release and commitfest schedules ===<br />
Dave P - Any proposed dates?<br />
Katz- We proposed a specific feature freeze date and no one objected<br />
Dave P - April 8?<br />
Katz- Yes<br />
Peter E- Can always change it next year if needed anyway<br />
Katz- Any objections?<br />
Tom- Official rather than unofficial end?<br />
Katz- Do we want to change anything?<br />
Peter G- Anything wonky could be addressed, seems to be pretty well worked out<br />
Katz- Can we try to shift more work to earlier? Time and human nature shows that we end up with things at the end but getting things in earlier<br />
Andres- better this year than last year, at least. Not sure if there's specific reasons but generally better.<br />
Munro- Do you mean buildfarm breakage?<br />
Andres- Buildfarm breakage but also people testing<br />
Noah- Reduce log jam by reducing uncertainty about when it ends, maybe other approaches are there too<br />
Katz- Having CI helps because it provides feedback more quickly<br />
Dave P- Haven't come up with schedule for next cycle yet<br />
Katz- Usual late q3/q4?, everyone is planning around that<br />
Dave P- Pick a date?<br />
Katz- Typically target one of the last two weeks in September<br />
Peter E- Usually end up in October but would like it to be earlier<br />
Noah- Which betas when to get there?<br />
Peter E- We don't usually plan these really and haven't ever really set up a date<br />
Joe- Seems like doing the same as we've done in the past, right?<br />
Dave P- Topic is are we going to change what we've done?<br />
Joe- July first CF for 17?<br />
Andres- Impression was that lots of review was happening but was a different set of people vs. people working on July CF<br />
Joe- When is RC?<br />
Katz- Generally September<br />
Tom- Don't have good sense of how much beta testing is happening<br />
Katz- We are still seeing bugs after the beta, which is normal, it's software and that happens, but more we get people to beta test the more we'll get figured out<br />
Dave P- That's a discussion we've had for every meeting, used to do a lot more alpha and beta but didn't seem to get people to actually test<br />
Katz- Uses say they are happy to help with testing but then they don't<br />
Peter G- Impression is that it's hardly the case that users test<br />
Dave P- Agreed, people don't tend to test<br />
Peter E- People expecting docker images and such but without that it ends up not happens<br />
Alex- Very inconvenient for some people to test<br />
Mark- EDB testing happens starting with betas because their product depends on it, but if they don't catch things then they release and having it go longer doesn't add anything really<br />
Peter E- There are others who do that, like PostGIS, where they don't do user testing but they test that their extensions work<br />
Dave P- pgAdmin doesn't test very early on<br />
Katz- Extensions do get tested early on though<br />
Andres- PostGIS actually keeps up pretty tightly<br />
Peter E- Other extensions don't really keep up with that<br />
Dave P- PostGIS watches for API breakage and such<br />
Andres- Yes, in their best interest to keep up<br />
Noah- What about pginfra?<br />
Magnus- Most of our pginfra is relitively small and simple<br />
Peter G- How would we know if more beta testing was happening?<br />
Tom- We'd have more bug reports<br />
Joe- Seems like we just need to set the dates out and then we change the plan if needed to, but with dates we actually have deadlines<br />
Dave P- Good point, users can then plan around that<br />
Katz- Typically the betas are around the same time each year, beta2 is around when we branch, beta3 is around August, then RC in September<br />
Tom- Did we do a beta right before branch?<br />
Katz- Usually, right in last week of June<br />
Peter E- Depends on open issues but once those are resolved we can release, looking at it now, 6 items, 2 are basically the same, could be done quick, last year had a super long list that took a lot longer to get there.<br />
Stephen- If we have a deadline then people can work towards it<br />
Peter E- Have a deadline or your feature gets reverted<br />
Heikki- Lets write down these deadlines so that people can work towards them<br />
Katz- Gives predictability to users so they can plan around it. How many people really go to .0? Suprisingly a lot of people do<br />
Joe- .0 is where the real testing starts.. lets just admit that and push to get that out<br />
Andres- Folks will just move to .1...<br />
Katz- There wasn't a lot of time between .0 and .1 and when there aren't a lot of bugs fixed between them then people feel comfortable<br />
Stephen- Should be close to the next quarterly release<br />
Katz- Think we agree, don't want too far before quarterly and don't want right after<br />
Dave P- September 14 is a Thursday, seems like a good date?<br />
Peter G- That seems fine<br />
Andres- Is there a chance we'd want to reduce the window between stamping and release, particularly when there are security issues? Seems to be long to have 4 days between those and packaging is faster.<br />
Tom- EDB didn't want to short it<br />
Dave P- Isn't packaging, it's more about QA and checking to make sure everything is ok and to have time to address them<br />
Peter G- Could adjust if needed<br />
Dave P- People know the schedule and have confidence that they can do things in time to test, et al<br />
Joe- Seems we could easily move it to Wednesday for the actual release process<br />
Andres- Do we actually have cases where things break between stamping and release?<br />
Dave P- Isn't always an issue with our stuff, sometimes it is with other things and with packaging<br />
Magnus- Those could be tested previously rather than after stamping<br />
Dave P- Could possibly do earlier but other libraries have new releases and such<br />
Magnus- RPMs and Debian systems get built but RPMs aren't really tested, but the Debian snapshots are actually tested<br />
Peter E- Can't start building the actual release packages until the actual release tarballs are published, need a bit of time in case things happen. Maybe could squeeze from Monday to Wednesday.<br />
Dave P- Wouldn't want to squeeze any more than Monday to Wednesday<br />
Heikki- If there is a hiccup with Debian or such then that is independent really<br />
Dave P- We release through our own servers and such really<br />
Peter E- We could go out and ask people about this and find out<br />
Noah- If we declare it earlier that it'll be a shorter release then people can make it work<br />
Joe- Folks don't really want to deploy things on a Friday, so it's better to bring it in earlier<br />
Michael P- Timezones should be considered too and Thursday in US is actually Friday on the other side of the world and people don't want to deploy on Fridays<br />
Dave P- Same for packagers really as many of them are in India<br />
Mark- Are people waiting until the last minute?<br />
Dave P- We don't really do snapshot builds because we depend on the tarballs<br />
Mark- If someone committed something 2 months before that breaks a build, you won't know?<br />
Dave P- We should know that but there's other libraries and things that are involved in the build which get updated as part of the installer and might break things and those aren't checked as often.<br />
Noah- Think of packaging as pipeline that's constantly getting built to test and make sure that things work<br />
Dave P- Requires people to spend time testing that things actually worked and having an automated system is an awful lot of work, particularly of GUI applications that have to work but are hard to test<br />
Heikki- Seems like consensus is Wednesday?<br />
Tom- We should ask the packagers list really as they are very involved<br />
Heikki- Yes, we should propose it to them and see what feedback we get.<br />
Dave P- Want to be careful with packages, we don't want to get into a position of forcing other folks to work 12-14 hour day because many people have this as a job and such.<br />
Noah- We should ask packagers what the impact ist.<br />
Katz- Need to give ample time to even consider this, should we test with beta releases?<br />
Peter E- Not a good test point because it doesn't require as much<br />
Dave P- Beta is also only one version vs. the regular releases<br />
Andres- Isn't the same pressure for the beta releases really either, no push on that<br />
Katz- Seems like we could just propose it for the next regular set of releases<br />
Dave P- We've blown our time and seems like we should move on<br />
Katz- Propose to packagers for November release to release on Wednesday instead<br />
Dave P- Sept 14 target for release?<br />
Katz- For v17 Sept 19, 2024<br />
<br />
==== Action items ====<br />
Propose to packagers moving to Wednesday for November release<br />
Propose to RMT September 14 2023 for v16 release<br />
Propose to have September 19 2024 for v17 release<br />
<br />
=== Renaming "master" branch to "main" ===<br />
Michael- This was proposed back in 2020 and was done by github. Git has changed itself to have an init default branch, also released in 2020. You can also set up a default branch. Andres committed a lot of changes to move to primary and to not use master/slave. How do people feel about the change of branch name?<br />
Peter E- I have a master's degree from university, what to do with that? Not sure that the term is really that objectionable<br />
Dave P- Have similar issue in pgAdmin with the "master" password, named for key that is used to unlock with other keys. Decided to leave as-is since there really wasn't another name.<br />
Peter E- That's also the general term usage in cryptography.<br />
Dave P- Wouldn't object to the change but not really excited about it.<br />
Heikki- Don't need to decide if it's a bad word or not but git and github have changed and should we maybe consider the change for that reason?<br />
Peter E- Git hasn't really changed it though<br />
Michael- Right, really only github has changed it<br />
Peter E- git from commandline vs. from github<br />
Peter G- Depends on the packaging maybe?<br />
Michael- Git config setting that can be changed<br />
Peter E- When this came up before, consensus was that we would consider changing when Git changes, but Git hasn't changed yet, so<br />
Noah- Think we said we would change when Git changes and Git has not changed yet.<br />
Peter E- One point was that if we don't change it then we'll have the same discussion every year and maybe we should change it just due to that. There's a bunch of extensions and tooling and such and this would create a lot of work for folks.<br />
Heikki- Doing a new git repo defaults to main on my laptop but that's because of configuration, but if you don't have that configuration set then you get a hint that if you don't have the value set that the default may change in the future.<br />
Dave P- What downside?<br />
Mark- No idea how many bugs will come out because of this with pipelines and such<br />
Andres- Is there a way to make it just work for users who have existing checked out repos using git sym refs or such?<br />
Peter E- People are expecting main to be there these days rather than master and we don't necessarily have to go purge everything of the specific term<br />
Noah- Key is to not surprise people too mucn, we don't want to be the last people to have that term<br />
Peter E- But then what about everything else like pgpool<br />
Noah- Feel that 'master' is less surprising than 'main' today but that's probably going to change moving forward.<br />
Heikki- Some extensions have already changed like pgrouting<br />
Tom- Maybe wait for git or Linux to change<br />
Mark- Poll the room, does anyone feel strongly one way or the other<br />
Tom- Kinda feel like its not worth the effort currently<br />
Michael- Not feeling like there is an urge to change given that git hasn't changed.<br />
Dave P- No one really strongly against it either<br />
Michael- Did it myself locally and was not hard to change. Also fine to just not do anything right now.<br />
Katz- Same conclusion as last time- wait for upstream or Linux to change.<br />
<br />
=== A brief PG15 RMT postmortem and what can we improve? ===<br />
Katz- Postmortem from RMT from last year. Unique things in v15 release which is useful to call out. RMT is Release Management Team. Prior to RMT we had releases drag on. Idea was to have a team who is able to push towards a release and make sure the system is stable and address open items. Two goals- release on time, and have as stable a release as possible. Some releases have been pushed into October but generally its been good and releases have been pretty stable but sometimes there are bugs that weren't caught until later minor releases. In general things have been good and stable but there's always opportunity to improve. SQL/JSON is an interesting case which was reverted late and it was a highlight feature. v15 ended up being a bit sparse in terms of major features. v16 has lots of awesome stuff on the other hand. SQL/JSON things which were challenging- highly anticipated feature even though its a lot of syntactic sugar, but other databases have sql/json and people want to move to PG. What made it challenging from an RMT perspective but it wasn't ready and people were still working on it but ultimately we had to revert it. Hard to tell someone to revert it but that's part of the job and it is what it is. With sql/json what the RMT was trying to do was see if we could get it in. Want feedback though, we still had a pretty stable release and we got the release on time more-or-less. Personally, I allowed going past deadlines which were set and that was a mistake because it ended up getting reverted late. Currently v15 seems to be pretty stable while there were some bugs. Are there things from that release and that effort that the RMT could benefit from hearing?<br />
Peter G- Think the RMT process has been effective and successful. All things considered the RMT does a good job. You mentioned the obvious big one for v15 but that was a pretty specific case.<br />
Andres- Wasn't that narrow but need to make sure the RMT isn't looking at things from a marketing perspective. Thinking about hey this is one of the major features.<br />
Peter G- How much time did the RMT give for that feature?<br />
Andres- Months, like 3 months, was a lot of time, really dragged on. Was also emotionally draining for people.<br />
Melanie- Is it intentional that what the RMT does is opaque?<br />
Peter G- Sort of.<br />
Peter E- Isn't really intentional to have it that way.<br />
Peter G- Think it was Noah that came up with the idea but was a good idea and is better to have a team who is informing someone that something has to be reverted rather than individuals going back and forth about it.<br />
Andres- Doesn't mean we can't document the process and the review.<br />
Peter G- There is a wiki page<br />
Andres- But how would you know to look for a RMT wiki page as a new contributor?<br />
Heikki- New contributors don't really need to worry about the RMT as it is more for committers to deal with typically.<br />
Melanie- Are all RMT communications open? Are there online meeting notes and such?<br />
Peter G- No<br />
Katz- RMT meetings we do try to document what we talk about and there's a google doc and things on hackers are public but notes are private but not typically anything earth shattering in those meeting notes.<br />
Peter G- Was on team but if there are off-the-record comments then those would be off-the-record either way and don't want people to take things personally.<br />
Melanie- Is there a list and a date for when things need to get reverted<br />
Peter G- The thing that is the real no-no is when someone isn't very communicative, but as long as there is an active discussion and an ongoing effort then things don't need to really have hard deadlines.<br />
Noah- First year or two, a couple of emails were sent to hackers to explain how the RMT works, could do that annually.<br />
Peter G- There is a email that was sent that sets up the RMT<br />
Noah- Model should follow what the supreme courts should be like and that they can have some private notes if they need to<br />
Mark- Is there an active issue? Is there hostility to RMT? If feature isn't ready that isn't really the RMT's fault.<br />
Andres- Would have been good to have been faster when it came to reverting what wasn't ready for v15. Dragging on wasn't good.<br />
Peter E- There wasn't urgency maybe because it seemed like it could be reverted at any time.<br />
Peter G- There were some tendrils where it reached out to that wouldn't have been good to keep, would have been good to pull it out sooner.<br />
Katz- Sometimes its important to just jump on a call and discuss it.<br />
Peter G- Point is, can you really expect to do it differently next time? Not sure that there is really a lesson. Maybe be more aggressive next time?<br />
Andres- Maybe lesson is to not allow the marketability of the feature to drive us to keep it longer than we should have.<br />
Katz- We don't want to risk our reputation for reliability<br />
Peter G- Objection is that it took a long time but ultimately it was reverted and that was the right decision and it was reached in time.<br />
Katz- One of my take-aways, maybe we should be targeting like beta2 to say if it's not ready by then, then it needs to go, so that everyone can kind of move on and can focus on stabilizing the release.<br />
Noah- More time you give people the more time people will take. Point is that things need to keep moving and there needs to be regular progress and if there's not then there has to be a really good reason. You also brought up if the RMT should do more testing?<br />
Katz- The goal is to make sure the release needs to be reliable but it should be a project goal to have it be reliable.<br />
Noah- Right, would be too much for the RMT to take on.<br />
<br />
==== Action items ====<br />
DO BETTER!<br />
When in doubt, use beta2 as the deadline.<br />
<br />
=== What are the big challenges for our users? What are the big challenges for us to solve? ===<br />
Katz- We have several days we will all be together, going to dive deep on a bunch of different things. What's nice about getting this group together is that we don't get to do this too often and focus and talk directly instead of being on laptops writing emails. Good to talk about higher-level challenges and not just technical ones. What are the biggest challenges for our users? From the users perspective, three buckets to try and put things in- availability, performance, emerging workloads. Good answers for HA. Biggest pain points- software upgrades, trying to get to zero downtime. Big users polled- whats biggest pain point- getting to zero downtime was #1 for all of them. Another big one was non-blocking schema changes. If there are exclusive locks being taken then it's practically the same as downtime.<br />
Peter G- Just knowing ahead of time what is going to happen<br />
Ketz- Another big theme is observability and be able to have predictability. Second is performance- trying to always get better performance, the direct i/o work, its going to take some time but there is a lot of excitement there. Also is vertical scalability where we can continue to grow up too. Saw systems with 24TB of RAM, but how much of that can PG really use? Also parallel operations. Finally emerging workloads- new topic that is really an old topic and old thing which has been around a long time, but vectors are a big thing these days and very hot and that's a good thing. Big arrays can push the limits of a page due to page size and how big the vector ends up being and people want to index these things and that can be a problem due to page size. What else is everyone hearing? Or project or other technical challenges? What are people hearing?<br />
Heikki- Connection pooling, max connections, can't change it after you start.. You need to set up pgbouncer but then that has challenges and limitations too.<br />
Katz- Yes, very good point.<br />
Andres- Getting HA is too hard and have to use a provider to get it. Trying to write down all the things needed you quickly find our tools really aren't good. Would be good to improve on that to make it realistic for users to have HA without a lot of effort.<br />
Peter E- Biggest challenge is that our development velocity isn't growing. Isn't shrinking either, but isn't growing. Has been pretty constant for like 10 years. Can use those resources to work on X or Y but if you want to work on something like major version upgrades which will take time every year forever and if you can't grow the pool then that takes a way a certain set of people.<br />
Jeff- For a lot of projects, the velocity reduces, so it isn't bad that we have been able to maintain.<br />
Andres- Agree with the concern, but not sure that the commits vs. lines changed is as much of a concern, but also the quality of code going in now is much higher vs. how it used to be, and if you look at the number of corruption bugs, that is much less now than it used to be.<br />
Peter E- Quality of code is going up but there's still a limit with the number of people and if we aren't growing the number of people then it's hard to grow new features really.<br />
Peter G- not that long ago there were much fewer (1?) person working full time on PG<br />
Katz- There is generally positive thing but we need to figure out a way to grow the pool.<br />
Andres- There is a danger when you have people full-time because new people won't be full time and we forget how weird things are in PG and makes it hard for new people to get into the community. Because we do it full time, we kind of forget that people have to get started somehow.<br />
Heikki- Definitely feel that, just hired someone new to work on PG, is kind of exciting to watch someone new learn about the cool things in PG but it's hard too because they have to figure out how to search the mailing list and such.<br />
Nathan- People do find the process very intimidating and the culture of the mailing list and as you describe it you end up thinking about how strange it is.<br />
Peter E- Maybe we over-document it and then things get out of date, and you can't delete wiki pages very easily(?). Now you can just git clone, hack, git format-patch and send it. Seems like maybe the wiki page has too much and should be made simpler.<br />
Stephen- Mentioned GSoC and introducing new people<br />
Heikki- Wasn't too bad to get new person<br />
Melanie- Getting people engaged and keeping them motivated can be really hard and people can get very frustrated with the process. Ultimate is to have people who are not full-time engaged and working on PG rather than just having full-time people. There are so many pieces that people have to understand. We aren't really helping new people and we aren't going to get new people if we don't put effort into this. I need to figure out a minimal repro for a bug, I need to figure out how to benchmark a given change. How to make those steps forward, even if they get detailed enough feedback ... I did a workshop recently on how to get feedback on a thread and how to take it and how to move forward with changes. Can't just get detailed feedback and then not be able to say "I don't know what to do next" but some people aren't really sure about how to do it. This group isn't as good at mentorship as we should be and we need to change our attitudes around that to try and be better.<br />
Katz- How do we actually do that, able to do it internally to a degree, but how do we do this in the community, but we need to put the time into it.<br />
Melanie- How I'm still here- worked at Pivotal and do pair-programming, spent hours working on things together, similar at Microsoft with Andres working together over time to get things figured out and understand but isn't necessarily always realistic.<br />
Amit- Is it possible to get people involved with smaller patches rather than getting new people in.<br />
Melanie- A lot of PG people suggest that and ask people to review code, but if you're in that place where it's hard for you to understand what's going on, so reviewing a patch can take a very very long time and it ends up being a big time committment and they're not sure what to do next.<br />
Katz- Much harder to provide reviews really<br />
Nathan- Reviewing in a way to move the patch forward can be really hard. Would spend days slugging through a given patch and that's really what you have to do to keep it going.<br />
Melanie- Not clear that having done that review that you get to keep working on PG, isn't really something you can necessarily go to your boss and show what you did.<br />
Andres- Sometimes you get involved and its really hard to figure out what is going on with people going in different directions sometimes and everyone sounds the same to someone new.<br />
Peter G- When I was getting going didn't feel that people, but was a different time too.<br />
Katz- How do we make it feel like it is more of a collaboration. We want new people to come and to contribute to PG.<br />
Melanie- If we had 10 more people who were super productive and be contributing and moving PG forward. We need new people who are doing small things too but we really need 10 more people who are really contributing a lot. If we could just all invest the time to find those people then we could move things forward very well.<br />
Peter G- Not sure how relevant it is that committers are actually good<br />
Mark- Sometimes people ask about getting beat up on list by Tom, but Tom was the one who told me what I was doing was wrong, and it ends up being a personality thing. Not everyone understands that feedback about patches need to be taken in a positive way.<br />
Melanie- It's about taking the time to respond to people, doesn't have to be super nice but it's about investing.<br />
Mark- Pair-programming does happen in the companies but it just does't happen on the list. Not sure how you could do it on the list.<br />
Heikki- To Melanie's point, we could be better about making sure to tell people what to do next when they are given feedback.<br />
Peter E- What was feedback that didn't help?<br />
Melanie- Feedback was like "not sure if this is how it should be, could you check that?" which wasn't very clear and also needed to provide a slightly different repro. Also, how to *prove* something is correct? Not sure how to do it.<br />
Peter E- Sometimes figuring out what the next steps are may actually be the job. If you have a thing which is "not sure if this correct, can you do more to show it is correct?"<br />
Melanie- People shy away from giving explicit instruction but that can actually be helpful to new people.<br />
Jeff- On a public list, don't really want to tell people to do specific things, but maybe a different channel would be good to help with that, maybe do it off-list.<br />
Heikki- Maybe phrase it differently<br />
Melanie- Hash join memory patch- off-list debugging was done, but more things were done on list and included things like "hey, to do this we should use these functions" or similar. Seems like people don't want to actually do that though even though it could be really helpful to new people.<br />
Mark- Looked at peoples patches and it is missing the test code and sometimes add it but then it seems like hijacking credit from people and maybe that isn't good.<br />
Heikki- Maybe that feels good to some people though<br />
Melanie- Others may like that as it means other people are involved in helping with maintenance of it going forward<br />
Peter G- If I don't know something then I don't really want to say it but sometimes want to say- hey have you thought about this differently, but that may not be helpful.<br />
Melanie- With specific feedback then that could help still<br />
Heikki- Another part of the problem is people submit patches and no one responds<br />
Mark- Sometimes people propose things that no one wants<br />
Andres- That does happen sometimes and the thread dies pretty quickly after just a couple of messages. If the patch seems reasonable but you don't want to commit it right now. Patches which are obviously wrong are easier to provide feedback on.<br />
Peter G- Ambiguity kills patches and try to avoid that.<br />
Melanie- It's about investing and you need to take the time to think about what the next steps are. The reason to take the time to do that is to grow the people even if the patch itself is not as interesting to you.<br />
Katz- Mentorship isn't going to happen on the list directly. Need to do it over chat or zoom or similar. Would be good to have some kind of organic process and it'll take time and it'll be more overhead.<br />
Heikki- As part of CF process, would be nice if there was a step where people are told what the next step is for everything. Maybe too much for the CFM?<br />
Peter E- Some CFMs do do that.<br />
Magnus- Doing that for 100+ patches takes an awful lot of time.<br />
Katz- Developer work is more valued but the project work is also absolutely critical and if need more people doing that then we should say that.<br />
Magnus- People are going to be more open to spending time mentoring people in the company vs. others who might get hired by competitors or other people.<br />
Melanie- Would be good to have a goal to respond to all patches, maybe as CFM or others<br />
Tom- No one wants to be speaking from the community, at least not without a long discussion<br />
Noah- Reviewers do do that though, speak from the community in some small way<br />
Jeff- A lot of orgs, projects are set up in a way that something will eventually get committed. If we are speaking as a committer, we can say this is what I would do next, but even as a committer, only some patches actually get accepted and committed and giving feedback may not mean that the patch will end up being committed.<br />
Joe- Would it make sense to have the CF have an option where someone can ask for a mentor?<br />
Peter E- Would be adding another job on top of what we already have going on, barely have time to review things<br />
Melanie- If you've not met that person in real life, it might be difficult to do<br />
Joe- Don't want to impose yourself on someone but if someone is asking for it and it could be off-list or non-public then it could work<br />
Katz- Maybe have some idea about having office hours or something<br />
Joe- Office hours, people don't end up showing up really<br />
Katz- Maybe we could figure out an async way to do it. Being able to just quickly ask someone about something can be really helpful.<br />
Melanie- One things with that, everyone who wants their patch to get in might ask for a mentor though. Maybe there could be some stage where you've been involved long enough and you want to be around, maybe you could have someone who could talk to you or be there to ask questions of.<br />
Noah- Maybe a box to check to just say that someone is open to off-list messages or such.<br />
Heikki- maybe at the end of the CF, the CFM could just email people and ask them if they know what the next steps are with a their patch.<br />
Magnus- Maybe a "need help" option in the CF?<br />
<br />
<br />
==== Action items ====<br />
<br />
<br />
<br />
=== High level thoughts and feedback on moving toward ICU as the preferred collation provider ===<br />
<br />
Jeff- Added a new wiki page which is linked, wanted to give general status info. Basically the situation is that we have problems with glibc. Benefits of ICU- platform independence and not depending on the OS for ordering. With ICU would be out of our hands but in the hands of the unicode org rather than it being from the OS which would be better. Independence also gives a bit more control over version changes. Really hard to control version of glibc, bit easier to control the version of ICU. Performance benefit also is good, can't use abbrev. keys with some versions of glibc.<br />
Peter G- Blanket assumption is all versions and that is a good assumption<br />
Jeff- More confidence in ICU for abbrev keys.<br />
Peter G- One might ask why you make that distinction and one reason is just historical and glibc doesn't have the same priorities as ICU does. The glibc folks didn't seem too concerned about the issues that they caused us, so why would it make sense to depend on glibc for such a critical thing. ICU is used by other projects who have similar requirements as PG and they seem to be looking at the problem in the same way that we do which is probably the most important thing. glibc doesn't seem to have the same sense of concern around it as we do.<br />
Stephen- Everyone here is generally agreed on ICU being better, I think?<br />
Jeff- One person not in the meeting today who has expressed some skepticism about it.<br />
Joe- ICU vs. glibc for collation is one topic, but another is how well is ICU integrated into PG, not enough people are really using it, it seems.<br />
Peter G- Definitely gotten easier in v15 to use ICU instead<br />
Joe- Is it integrated enough to make ICU the default is the real question<br />
Mark- Do we know that adopting ICU won't lead to corrupt indexes?<br />
Jeff- Changing versions of ICU can change sort ordering and therefore there can be corruption if you change the version of ICU.<br />
Peter G- One of the main goals of ICU is to have stability and to ensure that we don't end up with corrupted indexes. Of course ICU can have bugs too, but they take it very seriously if there are such cases.<br />
Joe- If you do an OS upgrade and have the ICU version change then that can cause problems, but you can install the matching version of ICU if needed unlike with glibc which can't really be changed due to everything else depending on it.<br />
Jeff- Need to be able to load multiple versions to address these things<br />
Dave P- Have used ICU in EPAS for a long time but it's an older version because it isn't easy to move people to a new version currently.<br />
Joe- Yes, need to work on that to be able move from one to another.<br />
Dave P- Assumes that there are multiple versions of ICU available for a given OS<br />
Peter G- Yes, that needs to be addressed too and is certainly a part of the problem.<br />
Noah- Not sure we see eye-to-eye with the ICU project in terms of having multiple versions around concurrently<br />
Peter G- Older versions of ICU definitely seem to be around and have continued to work for a very long time. Fundamentally you either have stability or don't, separate question about upgrade path, but a lot of people don't worry about that.<br />
Noah- Not just a packaging issue necessarily really<br />
Peter G- Can't really segregate the data from the code<br />
Dave P- No clean separation between code and lookup tables<br />
Joe- If you took new code and pointed it at the older tables you'd still end up with changes, doesn't really work. "copy-exact" came from intel / microchip world and idea is that everything is copied exactly and that's what you need to have to not break indexes<br />
Peter G- Fundamentally you have stability or you don't and don't really see that changing.<br />
Joe- In some performance testing, wrote collation torture test in python, one of the things noticed was in old versions of glibc it'll sort in 2 minutes or so in rhel7, with glibc 2.21 they pulled a cache out claiming it would help things and not hurt things, but took that 2m sort it went to 60m, but with ICU it's still 2m. Every modern version of glibc has this issue.<br />
Heikki- Asking the RMT, should we revert this change?<br />
Katz- From the RMT perspective, thinking of using beta2 as the cut-off, last week in June.<br />
Peter G- Making it the default signals intent, packagers aren't required to do things with that if they don't want to.<br />
Jeff- initdb has a default also which a packager doesn't really control<br />
Peter G- Would be good to have the option for people to opt-out of having it<br />
Katz- Packagers have some discretion with what the default is. ICU just focuses on collation, unlike glibc. Platform independence with ICU and consistency. Biggest area of concern is with upgrades because if you go from glibc with one major version to ICU in another major version, you have to rebuild all your indexes.<br />
Jeff- We maintain the provider across the major version upgrade<br />
Andres- pg_dump should really maintain collation/provider across.<br />
Peter E- Historically we don't maintain it through for providers or locale and such<br />
Dave P- This is going much deeper than we have time for.<br />
<br />
<br />
==== Action items ====<br />
<br />
<br />
<br />
=== Cloud operators have data of value to this community, e.g. frequency of errors that should never happen. What sort of data sharing regime might provide a good balance between value to the community and comfort for cloud operators? ===<br />
<br />
Noah- Get relatively little feedback from users. Clouds could possibly provide feedback. If the community thinks an error message never happens, but cloud operators do see it with some regularity. What would the community like from the cloud operators?<br />
Peter G- Have been doing this on some limited basis. Back-patched a case with VACUUM where throw an error which we don't really need to. Similar one in the last week. What's notable- wasn't really based on some kind of measure, just based on knowing that this happens in the field, at all, was the important part.<br />
Andres- think there is a lot of value in specific numbers, and if it changes over time (such as with crashes). A lot of errors that we throw that should never happens. Building list of errors that shouldn't ever happen but isn't easy to do. Would be nice to have a list of things that really *really* shouldn't happen.<br />
Katz- To the main question- talk to a lot of users and also $dayjob is a big users of PG itself. If had known that a given change had a big impact on upgrades might have done something a bit differently with it. $dayjob is generally open to sharing numbers and information with the community to help PG succeed and move forward, but need to know what data would be helpful to share.<br />
Nathan- Most of my contributions are from customers who are seeing things. Already happening from a development perspective. Fleet-wide data information can help with things that maybe should be pulled out/removed since no one is using it.<br />
Katz- Other thing is number one is "reduce my downtime" and can share that kind of info. People trying to move more and more workloads to PG.<br />
Noah- No shocks in terms of what seen from cloud run PG<br />
Peter G- Also haven't really gotten much in the way of shocks<br />
Melanie- Survey of how big are indexes and other things and was a paper/study that was published with lots of info. Also looked at how many people are using PL vs. other features.<br />
Peter G- Sometimes people have bias when it comes to what they think is being used and may not match.<br />
Katz- can see lots of edge cases that not other people see.<br />
Mark- If the cloud operators only operate the database in a certain way then only going to see certain bugs. Engineers seem to think corruption isn't often happening, but support deals with corruption all the time and people don't remember what different versions of PG they upgraded through, etc.<br />
Andres- Would be good to get stats for corruption, like what we have for checksum failures. Lots of errors where an error is thrown but we continue on. Increment some counter on corruption being detected.<br />
Melanie- Do any of the logging tools provide that?<br />
Stephen- Think pgbadger does? Could, certainly.<br />
Andres- Lot of things where we just don't see it because user can't reproduce it reliably. If you just know that you have this issue happening maybe could do something about it.<br />
Peter E- Could have stats for all error codes maybe<br />
Joe- Yeah, kind of what I was getting at.<br />
Andres- Many many are just error code internal though which isn't helpful.<br />
Peter E- Need to do better about that rather than just using internal.<br />
Peter E- Maybe make some error message sharable (no details included?)<br />
Andres- Have just a way to have the format string maybe<br />
Stephen- Maybe log line of code?<br />
Tom- Maybe the function name? Line might not be useful enough and changes<br />
Peter G- Just knowing that something is happening at all would be something and better.<br />
Andres- Anonymized stats could be useful, where could it go?<br />
Peter G- Some things might be sensitive commercially even as problems.<br />
Andres- Can't share some things but could share percentages and things<br />
Heikki- Would be interested to see if cases not expected to happen, but actually are happening.<br />
Katz- Would it help to have customers come to the mailing list?<br />
Heikki- Happy to share things too<br />
<br />
==== Action items ====<br />
<br />
<br />
<br />
=== Any other business ===<br />
<br />
Dave P- Challenge coins<br />
Joe- Packagers list include cloud providers?<br />
Tom- Would that be useful for them?<br />
Joe- Would be very helpful to know when things are coming<br />
Peter E- That's generally public though<br />
Joe- Don't believe we have a way to say that when we have a release tarball to tie it back to a specific commit which would be really good to have.<br />
Tom- The git commit hash does go into the tarball<br />
Joe- Can't test that really because of things that get generated being in the tarball. Would be nice if process for building tarball gave a way to tie it back.<br />
Michael- Debian reproducible builds might be something to look into<br />
<br />
Thomas- snapshot too old feature exists, doesn't work, no one is owning it, have talked about it a couple times.<br />
Tom- probably just need to kill it<br />
Peter E- Maybe leave the setting but ignore it<br />
Andres- Pretty invasive so not good to do in back-branches ... maybe just hard-code it to off. Wouldn't want to get rid of it in v16, but maybe v17<br />
Thomas- I can write a patch for v17<br />
<br />
LUNCH!<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=PgCon_2023_Developer_Meeting&diff=37914PgCon 2023 Developer Meeting2023-05-30T14:59:07Z<p>Sfrost: </p>
<hr />
<div>A meeting of the interested PostgreSQL developers is being planned for Tuesday 30 May, 2023 at the University of Ottawa, prior to pgCon 2023. In order to keep the numbers manageable, this meeting is by '''invitation only'''.<br />
Any questions regarding the invitations to this event should be directed to the team of individuals tasked with coming up with the list of people to invite:<br />
<br />
* Andres Freund<br />
* Stephen Frost<br />
* Dave Page<br />
<br />
An Unconference will be held on Friday for in-depth discussion of technical topics.<br />
<br />
This is a PostgreSQL Community event.<br />
<br />
== Meeting Goals ==<br />
<br />
* Define the schedule for the upcoming releases<br />
* Address any proposed timing, policy, or procedure issues<br />
* Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.<br />
* Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br />
<br />
== Time & Location ==<br />
<br />
The meeting will (probably) be:<br />
<br />
* 9:00AM to 12PM<br />
* DMS 3105 - Desmarais Hall, 55 Laurier Avenue East<br />
* University of Ottawa.<br />
<br />
Lunch will be served during the meeting.<br />
<br />
== COVID-19 ==<br />
<br />
The University of Ottawa's COVID-19 guidance can be found at https://www.uottawa.ca/en/covid-19. Wearing of masks at the Developer Meeting will be optional, however we do ask that people do not attend if they have COVID symptoms or have tested positive.<br />
<br />
== RSVPs ==<br />
<br />
The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!<br />
<br />
# Nathan Bossart<br />
# Joe Conway<br />
# Jeff Davis<br />
# Mark Dilger<br />
# Peter Eisentraut<br />
# Andres Freund<br />
# Stephen Frost<br />
# Etsuro Fujita<br />
# Peter Geoghegan<br />
# Magnus Hagander<br />
# Amit Kapila<br />
# Jonathan Katz<br />
# Alexander Kukushkin<br />
# Tom Lane<br />
# Heikki Linnakangas<br />
# Noah Misch<br />
# Thomas Munro<br />
# Dave Page<br />
# Michael Paquier<br />
# Melanie Plageman<br />
# Masahiko Sawada<br />
# Peter Smith<br />
# Tomas Vondra<br />
<br />
The following people will not be in Ottawa, and do not plan to attend:<br />
<br />
# Masao Fujii<br />
# Daniel Gustafsson<br />
# Álvaro Herrera<br />
# Tatsuo Ishii<br />
# Alexander Korotkov<br />
# Amit Langote<br />
# Dean Rasheed<br />
# David Rowley<br />
<br />
== Agenda Items ==<br />
''Please add suggestions for agenda items here. (with your name)''<br />
<br />
* 16.0 release and commitfest schedule (Dave)<br />
* Renaming "master" branch to "main"? (Michael)<br />
* A brief PG15 RMT postmortem and what can we improve? (Jonathan)<br />
* What are the big challenges for our users? What are the big challenges for us to solve? (Jonathan)<br />
* Cloud operators have data of value to this community, e.g. frequency of errors that should never happen. What sort of data sharing regime might provide a good balance between value to the community and comfort for cloud operators? (Noah)<br />
* High level thoughts and feedback on moving toward ICU as the preferred collation provider (Jeff)<br />
<br />
==Agenda==<br />
<br />
{| border="1" cellpadding="4" cellspacing="0"<br />
!Time<br />
!Item<br />
!Presenter<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|09:00 - 09:10<br />
|Welcome and introductions<br />
|Dave Page<br />
<br />
|- <br />
|09:10 - 09:20<br />
|Release and commitfest schedules<br />
|Dave Page<br />
<br />
|- <br />
|09:20 - 09:35<br />
|Renaming "master" branch to "main"<br />
|Michael Paquier<br />
<br />
|- <br />
|09:35 - 10:00<br />
|A brief PG15 RMT postmortem and what can we improve?<br />
|Jonathan Katz<br />
<br />
|- <br />
|10:00 - 10:30<br />
|What are the big challenges for our users? What are the big challenges for us to solve?<br />
|Jonathan Katz<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|10:30 - 11:00<br />
|Coffee break<br />
|All<br />
<br />
|- <br />
|11:00 - 11:20<br />
|High level thoughts and feedback on moving toward ICU as the preferred collation provider [[StateOfICU]]<br />
|Jeff Davis<br />
<br />
|- <br />
|11:20 - 11:50<br />
|Cloud operators have data of value to this community, e.g. frequency of errors that should never happen. What sort of data sharing regime might provide a good balance between value to the community and comfort for cloud operators? <br />
|Noah Misch<br />
<br />
|- <br />
|11:50 - 12:00<br />
|Any other business<br />
|Dave Page<br />
<br />
|- style="font-style:italic;background-color:lightgray;"<br />
|12:00<br />
|Lunch<br />
|<br />
<br />
|}<br />
<br />
Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move materially however). Any remaining time before lunch may be used for Commitfest item triage or other activities.<br />
<br />
== Minutes ==<br />
Stephen Frost recording<br />
<br />
=== Welcome and Introductions ===<br />
Dave Page - General opening<br />
Introductions<br />
<br />
=== Release and commitfest schedules ===<br />
Dave P - Any proposed dates?<br />
Katz- We proposed a specific feature freeze date and no one objected<br />
Dave P - April 8?<br />
Katz- Yes<br />
Peter E- Can always change it next year if needed anyway<br />
Katz- Any objections?<br />
Tom- Official rather than unofficial end?<br />
Katz- Do we want to change anything?<br />
Peter G- Anything wonky could be addressed, seems to be pretty well worked out<br />
Katz- Can we try to shift more work to earlier? Time and human nature shows that we end up with things at the end but getting things in earlier<br />
Andres- better this year than last year, at least. Not sure if there's specific reasons but generally better.<br />
Munro- Do you mean buildfarm breakage?<br />
Andres- Buildfarm breakage but also people testing<br />
Noah- Reduce log jam by reducing uncertainty about when it ends, maybe other approaches are there too<br />
Katz- Having CI helps because it provides feedback more quickly<br />
Dave P- Haven't come up with schedule for next cycle yet<br />
Katz- Usual late q3/q4?, everyone is planning around that<br />
Dave P- Pick a date?<br />
Katz- Typically target one of the last two weeks in September<br />
Peter E- Usually end up in October but would like it to be earlier<br />
Noah- Which betas when to get there?<br />
Peter E- We don't usually plan these really and haven't ever really set up a date<br />
Joe- Seems like doing the same as we've done in the past, right?<br />
Dave P- Topic is are we going to change what we've done?<br />
Joe- July first CF for 17?<br />
Andres- Impression was that lots of review was happening but was a different set of people vs. people working on July CF<br />
Joe- When is RC?<br />
Katz- Generally September<br />
Tom- Don't have good sense of how much beta testing is happening<br />
Katz- We are still seeing bugs after the beta, which is normal, it's software and that happens, but more we get people to beta test the more we'll get figured out<br />
Dave P- That's a discussion we've had for every meeting, used to do a lot more alpha and beta but didn't seem to get people to actually test<br />
Katz- Uses say they are happy to help with testing but then they don't<br />
Peter G- Impression is that it's hardly the case that users test<br />
Dave P- Agreed, people don't tend to test<br />
Peter E- People expecting docker images and such but without that it ends up not happens<br />
Alex- Very inconvenient for some people to test<br />
Mark- EDB testing happens starting with betas because their product depends on it, but if they don't catch things then they release and having it go longer doesn't add anything really<br />
Peter E- There are others who do that, like PostGIS, where they don't do user testing but they test that their extensions work<br />
Dave P- pgAdmin doesn't test very early on<br />
Katz- Extensions do get tested early on though<br />
Andres- PostGIS actually keeps up pretty tightly<br />
Peter E- Other extensions don't really keep up with that<br />
Dave P- PostGIS watches for API breakage and such<br />
Andres- Yes, in their best interest to keep up<br />
Noah- What about pginfra?<br />
Magnus- Most of our pginfra is relitively small and simple<br />
Peter G- How would we know if more beta testing was happening?<br />
Tom- We'd have more bug reports<br />
Joe- Seems like we just need to set the dates out and then we change the plan if needed to, but with dates we actually have deadlines<br />
Dave P- Good point, users can then plan around that<br />
Katz- Typically the betas are around the same time each year, beta2 is around when we branch, beta3 is around August, then RC in September<br />
Tom- Did we do a beta right before branch?<br />
Katz- Usually, right in last week of June<br />
Peter E- Depends on open issues but once those are resolved we can release, looking at it now, 6 items, 2 are basically the same, could be done quick, last year had a super long list that took a lot longer to get there.<br />
Stephen- If we have a deadline then people can work towards it<br />
Peter E- Have a deadline or your feature gets reverted<br />
Heikki- Lets write down these deadlines so that people can work towards them<br />
Katz- Gives predictability to users so they can plan around it. How many people really go to .0? Suprisingly a lot of people do<br />
Joe- .0 is where the real testing starts.. lets just admit that and push to get that out<br />
Andres- Folks will just move to .1...<br />
Katz- There wasn't a lot of time between .0 and .1 and when there aren't a lot of bugs fixed between them then people feel comfortable<br />
Stephen- Should be close to the next quarterly release<br />
Katz- Think we agree, don't want too far before quarterly and don't want right after<br />
Dave P- September 14 is a Thursday, seems like a good date?<br />
Peter G- That seems fine<br />
Andres- Is there a chance we'd want to reduce the window between stamping and release, particularly when there are security issues? Seems to be long to have 4 days between those and packaging is faster.<br />
Tom- EDB didn't want to short it<br />
Dave P- Isn't packaging, it's more about QA and checking to make sure everything is ok and to have time to address them<br />
Peter G- Could adjust if needed<br />
Dave P- People know the schedule and have confidence that they can do things in time to test, et al<br />
Joe- Seems we could easily move it to Wednesday for the actual release process<br />
Andres- Do we actually have cases where things break between stamping and release?<br />
Dave P- Isn't always an issue with our stuff, sometimes it is with other things and with packaging<br />
Magnus- Those could be tested previously rather than after stamping<br />
Dave P- Could possibly do earlier but other libraries have new releases and such<br />
Magnus- RPMs and Debian systems get built but RPMs aren't really tested, but the Debian snapshots are actually tested<br />
Peter E- Can't start building the actual release packages until the actual release tarballs are published, need a bit of time in case things happen. Maybe could squeeze from Monday to Wednesday.<br />
Dave P- Wouldn't want to squeeze any more than Monday to Wednesday<br />
Heikki- If there is a hiccup with Debian or such then that is independent really<br />
Dave P- We release through our own servers and such really<br />
Peter E- We could go out and ask people about this and find out<br />
Noah- If we declare it earlier that it'll be a shorter release then people can make it work<br />
Joe- Folks don't really want to deploy things on a Friday, so it's better to bring it in earlier<br />
Michael P- Timezones should be considered too and Thursday in US is actually Friday on the other side of the world and people don't want to deploy on Fridays<br />
Dave P- Same for packagers really as many of them are in India<br />
Mark- Are people waiting until the last minute?<br />
Dave P- We don't really do snapshot builds because we depend on the tarballs<br />
Mark- If someone committed something 2 months before that breaks a build, you won't know?<br />
Dave P- We should know that but there's other libraries and things that are involved in the build which get updated as part of the installer and might break things and those aren't checked as often.<br />
Noah- Think of packaging as pipeline that's constantly getting built to test and make sure that things work<br />
Dave P- Requires people to spend time testing that things actually worked and having an automated system is an awful lot of work, particularly of GUI applications that have to work but are hard to test<br />
Heikki- Seems like consensus is Wednesday?<br />
Tom- We should ask the packagers list really as they are very involved<br />
Heikki- Yes, we should propose it to them and see what feedback we get.<br />
Dave P- Want to be careful with packages, we don't want to get into a position of forcing other folks to work 12-14 hour day because many people have this as a job and such.<br />
Noah- We should ask packagers what the impact ist.<br />
Katz- Need to give ample time to even consider this, should we test with beta releases?<br />
Peter E- Not a good test point because it doesn't require as much<br />
Dave P- Beta is also only one version vs. the regular releases<br />
Andres- Isn't the same pressure for the beta releases really either, no push on that<br />
Katz- Seems like we could just propose it for the next regular set of releases<br />
Dave P- We've blown our time and seems like we should move on<br />
Katz- Propose to packagers for November release to release on Wednesday instead<br />
Dave P- Sept 14 target for release?<br />
Katz- For v17 Sept 19, 2024<br />
<br />
==== Action items ====<br />
Propose to packagers moving to Wednesday for November release<br />
Propose to RMT September 14 2023 for v16 release<br />
Propose to have September 19 2024 for v17 release<br />
<br />
=== Renaming "master" branch to "main" ===<br />
Michael- This was proposed back in 2020 and was done by github. Git has changed itself to have an init default branch, also released in 2020. You can also set up a default branch. Andres committed a lot of changes to move to primary and to not use master/slave. How do people feel about the change of branch name?<br />
Peter E- I have a master's degree from university, what to do with that? Not sure that the term is really that objectionable<br />
Dave P- Have similar issue in pgAdmin with the "master" password, named for key that is used to unlock with other keys. Decided to leave as-is since there really wasn't another name.<br />
Peter E- That's also the general term usage in cryptography.<br />
Dave P- Wouldn't object to the change but not really excited about it.<br />
Heikki- Don't need to decide if it's a bad word or not but git and github have changed and should we maybe consider the change for that reason?<br />
Peter E- Git hasn't really changed it though<br />
Michael- Right, really only github has changed it<br />
Peter E- git from commandline vs. from github<br />
Peter G- Depends on the packaging maybe?<br />
Michael- Git config setting that can be changed<br />
Peter E- When this came up before, consensus was that we would consider changing when Git changes, but Git hasn't changed yet, so<br />
Noah- Think we said we would change when Git changes and Git has not changed yet.<br />
Peter E- One point was that if we don't change it then we'll have the same discussion every year and maybe we should change it just due to that. There's a bunch of extensions and tooling and such and this would create a lot of work for folks.<br />
Heikki- Doing a new git repo defaults to main on my laptop but that's because of configuration, but if you don't have that configuration set then you get a hint that if you don't have the value set that the default may change in the future.<br />
Dave P- What downside?<br />
Mark- No idea how many bugs will come out because of this with pipelines and such<br />
Andres- Is there a way to make it just work for users who have existing checked out repos using git sym refs or such?<br />
Peter E- People are expecting main to be there these days rather than master and we don't necessarily have to go purge everything of the specific term<br />
Noah- Key is to not surprise people too mucn, we don't want to be the last people to have that term<br />
Peter E- But then what about everything else like pgpool<br />
Noah- Feel that 'master' is less surprising than 'main' today but that's probably going to change moving forward.<br />
Heikki- Some extensions have already changed like pgrouting<br />
Tom- Maybe wait for git or Linux to change<br />
Mark- Poll the room, does anyone feel strongly one way or the other<br />
Tom- Kinda feel like its not worth the effort currently<br />
Michael- Not feeling like there is an urge to change given that git hasn't changed.<br />
Dave P- No one really strongly against it either<br />
Michael- Did it myself locally and was not hard to change. Also fine to just not do anything right now.<br />
Katz- Same conclusion as last time- wait for upstream or Linux to change.<br />
<br />
=== A brief PG15 RMT postmortem and what can we improve? ===<br />
Katz- Postmortem from RMT from last year. Unique things in v15 release which is useful to call out. RMT is Release Management Team. Prior to RMT we had releases drag on. Idea was to have a team who is able to push towards a release and make sure the system is stable and address open items. Two goals- release on time, and have as stable a release as possible. Some releases have been pushed into October but generally its been good and releases have been pretty stable but sometimes there are bugs that weren't caught until later minor releases. In general things have been good and stable but there's always opportunity to improve. SQL/JSON is an interesting case which was reverted late and it was a highlight feature. v15 ended up being a bit sparse in terms of major features. v16 has lots of awesome stuff on the other hand. SQL/JSON things which were challenging- highly anticipated feature even though its a lot of syntactic sugar, but other databases have sql/json and people want to move to PG. What made it challenging from an RMT perspective but it wasn't ready and people were still working on it but ultimately we had to revert it. Hard to tell someone to revert it but that's part of the job and it is what it is. With sql/json what the RMT was trying to do was see if we could get it in. Want feedback though, we still had a pretty stable release and we got the release on time more-or-less. Personally, I allowed going past deadlines which were set and that was a mistake because it ended up getting reverted late. Currently v15 seems to be pretty stable while there were some bugs. Are there things from that release and that effort that the RMT could benefit from hearing?<br />
Peter G- Think the RMT process has been effective and successful. All things considered the RMT does a good job. You mentioned the obvious big one for v15 but that was a pretty specific case.<br />
Andres- Wasn't that narrow but need to make sure the RMT isn't looking at things from a marketing perspective. Thinking about hey this is one of the major features.<br />
Peter G- How much time did the RMT give for that feature?<br />
Andres- Months, like 3 months, was a lot of time, really dragged on. Was also emotionally draining for people.<br />
Melanie- Is it intentional that what the RMT does is opaque?<br />
Peter G- Sort of.<br />
Peter E- Isn't really intentional to have it that way.<br />
Peter G- Think it was Noah that came up with the idea but was a good idea and is better to have a team who is informing someone that something has to be reverted rather than individuals going back and forth about it.<br />
Andres- Doesn't mean we can't document the process and the review.<br />
Peter G- There is a wiki page<br />
Andres- But how would you know to look for a RMT wiki page as a new contributor?<br />
Heikki- New contributors don't really need to worry about the RMT as it is more for committers to deal with typically.<br />
Melanie- Are all RMT communications open? Are there online meeting notes and such?<br />
Peter G- No<br />
Katz- RMT meetings we do try to document what we talk about and there's a google doc and things on hackers are public but notes are private but not typically anything earth shattering in those meeting notes.<br />
Peter G- Was on team but if there are off-the-record comments then those would be off-the-record either way and don't want people to take things personally.<br />
Melanie- Is there a list and a date for when things need to get reverted<br />
Peter G- The thing that is the real no-no is when someone isn't very communicative, but as long as there is an active discussion and an ongoing effort then things don't need to really have hard deadlines.<br />
Noah- First year or two, a couple of emails were sent to hackers to explain how the RMT works, could do that annually.<br />
Peter G- There is a email that was sent that sets up the RMT<br />
Noah- Model should follow what the supreme courts should be like and that they can have some private notes if they need to<br />
Mark- Is there an active issue? Is there hostility to RMT? If feature isn't ready that isn't really the RMT's fault.<br />
Andres- Would have been good to have been faster when it came to reverting what wasn't ready for v15. Dragging on wasn't good.<br />
Peter E- There wasn't urgency maybe because it seemed like it could be reverted at any time.<br />
Peter G- There were some tendrils where it reached out to that wouldn't have been good to keep, would have been good to pull it out sooner.<br />
Katz- Sometimes its important to just jump on a call and discuss it.<br />
Peter G- Point is, can you really expect to do it differently next time? Not sure that there is really a lesson. Maybe be more aggressive next time?<br />
Andres- Maybe lesson is to not allow the marketability of the feature to drive us to keep it longer than we should have.<br />
Katz- We don't want to risk our reputation for reliability<br />
Peter G- Objection is that it took a long time but ultimately it was reverted and that was the right decision and it was reached in time.<br />
Katz- One of my take-aways, maybe we should be targeting like beta2 to say if it's not ready by then, then it needs to go, so that everyone can kind of move on and can focus on stabilizing the release.<br />
Noah- More time you give people the more time people will take. Point is that things need to keep moving and there needs to be regular progress and if there's not then there has to be a really good reason. You also brought up if the RMT should do more testing?<br />
Katz- The goal is to make sure the release needs to be reliable but it should be a project goal to have it be reliable.<br />
Noah- Right, would be too much for the RMT to take on.<br />
<br />
==== Action items ====<br />
DO BETTER!<br />
When in doubt, use beta2 as the deadline.<br />
<br />
=== What are the big challenges for our users? What are the big challenges for us to solve? ===<br />
Katz- We have several days we will all be together, going to dive deep on a bunch of different things. What's nice about getting this group together is that we don't get to do this too often and focus and talk directly instead of being on laptops writing emails. Good to talk about higher-level challenges and not just technical ones. What are the biggest challenges for our users? From the users perspective, three buckets to try and put things in- availability, performance, emerging workloads. Good answers for HA. Biggest pain points- software upgrades, trying to get to zero downtime. Big users polled- whats biggest pain point- getting to zero downtime was #1 for all of them. Another big one was non-blocking schema changes. If there are exclusive locks being taken then it's practically the same as downtime.<br />
Peter G- Just knowing ahead of time what is going to happen<br />
Ketz- Another big theme is observability and be able to have predictability. Second is performance- trying to always get better performance, the direct i/o work, its going to take some time but there is a lot of excitement there. Also is vertical scalability where we can continue to grow up too. Saw systems with 24TB of RAM, but how much of that can PG really use? Also parallel operations. Finally emerging workloads- new topic that is really an old topic and old thing which has been around a long time, but vectors are a big thing these days and very hot and that's a good thing. Big arrays can push the limits of a page due to page size and how big the vector ends up being and people want to index these things and that can be a problem due to page size. What else is everyone hearing? Or project or other technical challenges? What are people hearing?<br />
Heikki- Connection pooling, max connections, can't change it after you start.. You need to set up pgbouncer but then that has challenges and limitations too.<br />
Katz- Yes, very good point.<br />
Andres- Getting HA is too hard and have to use a provider to get it. Trying to write down all the things needed you quickly find our tools really aren't good. Would be good to improve on that to make it realistic for users to have HA without a lot of effort.<br />
Peter E- Biggest challenge is that our development velocity isn't growing. Isn't shrinking either, but isn't growing. Has been pretty constant for like 10 years. Can use those resources to work on X or Y but if you want to work on something like major version upgrades which will take time every year forever and if you can't grow the pool then that takes a way a certain set of people.<br />
Jeff- For a lot of projects, the velocity reduces, so it isn't bad that we have been able to maintain.<br />
Andres- Agree with the concern, but not sure that the commits vs. lines changed is as much of a concern, but also the quality of code going in now is much higher vs. how it used to be, and if you look at the number of corruption bugs, that is much less now than it used to be.<br />
Peter E- Quality of code is going up but there's still a limit with the number of people and if we aren't growing the number of people then it's hard to grow new features really.<br />
Peter G- not that long ago there were much fewer (1?) person working full time on PG<br />
Katz- There is generally positive thing but we need to figure out a way to grow the pool.<br />
Andres- There is a danger when you have people full-time because new people won't be full time and we forget how weird things are in PG and makes it hard for new people to get into the community. Because we do it full time, we kind of forget that people have to get started somehow.<br />
Heikki- Definitely feel that, just hired someone new to work on PG, is kind of exciting to watch someone new learn about the cool things in PG but it's hard too because they have to figure out how to search the mailing list and such.<br />
Nathan- People do find the process very intimidating and the culture of the mailing list and as you describe it you end up thinking about how strange it is.<br />
Peter E- Maybe we over-document it and then things get out of date, and you can't delete wiki pages very easily(?). Now you can just git clone, hack, git format-patch and send it. Seems like maybe the wiki page has too much and should be made simpler.<br />
Stephen- Mentioned GSoC and introducing new people<br />
Heikki- Wasn't too bad to get new person<br />
Melanie- Getting people engaged and keeping them motivated can be really hard and people can get very frustrated with the process. Ultimate is to have people who are not full-time engaged and working on PG rather than just having full-time people. There are so many pieces that people have to understand. We aren't really helping new people and we aren't going to get new people if we don't put effort into this. I need to figure out a minimal repro for a bug, I need to figure out how to benchmark a given change. How to make those steps forward, even if they get detailed enough feedback ... I did a workshop recently on how to get feedback on a thread and how to take it and how to move forward with changes. Can't just get detailed feedback and then not be able to say "I don't know what to do next" but some people aren't really sure about how to do it. This group isn't as good at mentorship as we should be and we need to change our attitudes around that to try and be better.<br />
Katz- How do we actually do that, able to do it internally to a degree, but how do we do this in the community, but we need to put the time into it.<br />
Melanie- How I'm still here- worked at Pivotal and do pair-programming, spent hours working on things together, similar at Microsoft with Andres working together over time to get things figured out and understand but isn't necessarily always realistic.<br />
Amit- Is it possible to get people involved with smaller patches rather than getting new people in.<br />
Melanie- A lot of PG people suggest that and ask people to review code, but if you're in that place where it's hard for you to understand what's going on, so reviewing a patch can take a very very long time and it ends up being a big time committment and they're not sure what to do next.<br />
Katz- Much harder to provide reviews really<br />
Nathan- Reviewing in a way to move the patch forward can be really hard. Would spend days slugging through a given patch and that's really what you have to do to keep it going.<br />
Melanie- Not clear that having done that review that you get to keep working on PG, isn't really something you can necessarily go to your boss and show what you did.<br />
Andres- Sometimes you get involved and its really hard to figure out what is going on with people going in different directions sometimes and everyone sounds the same to someone new.<br />
Peter G- When I was getting going didn't feel that people, but was a different time too.<br />
Katz- How do we make it feel like it is more of a collaboration. We want new people to come and to contribute to PG.<br />
Melanie- If we had 10 more people who were super productive and be contributing and moving PG forward. We need new people who are doing small things too but we really need 10 more people who are really contributing a lot. If we could just all invest the time to find those people then we could move things forward very well.<br />
Peter G- Not sure how relevant it is that committers are actually good<br />
Mark- Sometimes people ask about getting beat up on list by Tom, but Tom was the one who told me what I was doing was wrong, and it ends up being a personality thing. Not everyone understands that feedback about patches need to be taken in a positive way.<br />
Melanie- It's about taking the time to respond to people, doesn't have to be super nice but it's about investing.<br />
Mark- Pair-programming does happen in the companies but it just does't happen on the list. Not sure how you could do it on the list.<br />
Heikki- To Melanie's point, we could be better about making sure to tell people what to do next when they are given feedback.<br />
Peter E- What was feedback that didn't help?<br />
Melanie- Feedback was like "not sure if this is how it should be, could you check that?" which wasn't very clear and also needed to provide a slightly different repro. Also, how to *prove* something is correct? Not sure how to do it.<br />
Peter E- Sometimes figuring out what the next steps are may actually be the job. If you have a thing which is "not sure if this correct, can you do more to show it is correct?"<br />
Melanie- People shy away from giving explicit instruction but that can actually be helpful to new people.<br />
Jeff- On a public list, don't really want to tell people to do specific things, but maybe a different channel would be good to help with that, maybe do it off-list.<br />
Heikki- Maybe phrase it differently<br />
Melanie- Hash join memory patch- off-list debugging was done, but more things were done on list and included things like "hey, to do this we should use these functions" or similar. Seems like people don't want to actually do that though even though it could be really helpful to new people.<br />
Mark- Looked at peoples patches and it is missing the test code and sometimes add it but then it seems like hijacking credit from people and maybe that isn't good.<br />
Heikki- Maybe that feels good to some people though<br />
Melanie- Others may like that as it means other people are involved in helping with maintenance of it going forward<br />
Peter G- If I don't know something then I don't really want to say it but sometimes want to say- hey have you thought about this differently, but that may not be helpful.<br />
Melanie- With specific feedback then that could help still<br />
Heikki- Another part of the problem is people submit patches and no one responds<br />
Mark- Sometimes people propose things that no one wants<br />
Andres- That does happen sometimes and the thread dies pretty quickly after just a couple of messages. If the patch seems reasonable but you don't want to commit it right now. Patches which are obviously wrong are easier to provide feedback on.<br />
Peter G- Ambiguity kills patches and try to avoid that.<br />
Melanie- It's about investing and you need to take the time to think about what the next steps are. The reason to take the time to do that is to grow the people even if the patch itself is not as interesting to you.<br />
Katz- Mentorship isn't going to happen on the list directly. Need to do it over chat or zoom or similar. Would be good to have some kind of organic process and it'll take time and it'll be more overhead.<br />
Heikki- As part of CF process, would be nice if there was a step where people are told what the next step is for everything. Maybe too much for the CFM?<br />
Peter E- Some CFMs do do that.<br />
Magnus- Doing that for 100+ patches takes an awful lot of time.<br />
Katz- Developer work is more valued but the project work is also absolutely critical and if need more people doing that then we should say that.<br />
Magnus- People are going to be more open to spending time mentoring people in the company vs. others who might get hired by competitors or other people.<br />
Melanie- Would be good to have a goal to respond to all patches, maybe as CFM or others<br />
Tom- No one wants to be speaking from the community, at least not without a long discussion<br />
Noah- Reviewers do do that though, speak from the community in some small way<br />
Jeff- A lot of orgs, projects are set up in a way that something will eventually get committed. If we are speaking as a committer, we can say this is what I would do next, but even as a committer, only some patches actually get accepted and committed and giving feedback may not mean that the patch will end up being committed.<br />
Joe- Would it make sense to have the CF have an option where someone can ask for a mentor?<br />
Peter E- Would be adding another job on top of what we already have going on, barely have time to review things<br />
Melanie- If you've not met that person in real life, it might be difficult to do<br />
Joe- Don't want to impose yourself on someone but if someone is asking for it and it could be off-list or non-public then it could work<br />
Katz- Maybe have some idea about having office hours or something<br />
Joe- Office hours, people don't end up showing up really<br />
Katz- Maybe we could figure out an async way to do it. Being able to just quickly ask someone about something can be really helpful.<br />
Melanie- One things with that, everyone who wants their patch to get in might ask for a mentor though. Maybe there could be some stage where you've been involved long enough and you want to be around, maybe you could have someone who could talk to you or be there to ask questions of.<br />
Noah- Maybe a box to check to just say that someone is open to off-list messages or such.<br />
Heikki- maybe at the end of the CF, the CFM could just email people and ask them if they know what the next steps are with a their patch.<br />
Magnus- Maybe a "need help" option in the CF?<br />
<br />
<br />
==== Action items ====<br />
<br />
<br />
<br />
=== High level thoughts and feedback on moving toward ICU as the preferred collation provider ===<br />
=== Cloud operators have data of value to this community, e.g. frequency of errors that should never happen. What sort of data sharing regime might provide a good balance between value to the community and comfort for cloud operators? ===<br />
=== Any other business ===<br />
<br />
[[Category:Developer Meeting]]</div>Sfrosthttps://wiki.postgresql.org/index.php?title=Committers&diff=37775Committers2023-04-21T14:50:40Z<p>Sfrost: </p>
<hr />
<div>This is the current list of people with access to push to the git repository with their user names. For technical details on how committing works, see [[Committing with Git]]. Note: This is just a list of people who currently have access to push to git; for information on current and previous contributors, see the [http://www.postgresql.org/community/contributors/ contributor profiles] section of the web site. Note: The names are listed here in order of first commit or when added as a committer, oldest first; this isn't intended to imply anything about depth of contribution.<br />
<br />
* Bruce Momjian (momjian)<br />
* Tom Lane (tgl)<br />
* Michael Meskes (meskes)<br />
* Tatsuo Ishii (ishii)<br />
* Peter Eisentraut (petere)<br />
* Joe Conway (joe)<br />
* Alvaro Herrera (alvherre)<br />
* Andrew Dunstan (adunstan)<br />
* Magnus Hagander (mha)<br />
* Heikki Linnakangas (heikki)<br />
* Robert Haas (rhaas)<br />
* Jeff Davis (jdavis)<br />
* Stephen Frost (sfrost)<br />
* Fujii Masao (fujii)<br />
* Noah Misch (noah)<br />
* Andres Freund (andres)<br />
* Dean Rasheed (deanr)<br />
* Andrew Gierth (rhodiumtoad)<br />
* Alexander Korotkov (smagen)<br />
* Amit Kapila (amitkapila)<br />
* Tomas Vondra (fuzzycz)<br />
* Michael Paquier (michael-kun)<br />
* Thomas Munro (macdice)<br />
* Peter Geoghegan (pgeoghegan)<br />
* Etsuro Fujita (efujita)<br />
* David Rowley (drowley)<br />
* Daniel Gustafsson (d_gustafsson)<br />
* John Naylor (john.naylor)<br />
* Nathan Bossart (nbossart)<br />
* Amit Langote (amitlan)<br />
* Masahiko Sawada (msawada)<br />
<br />
== Notes on the Commit Log ==<br />
<br />
Hundreds of developers have successfully contributed work to PostgreSQL over more than 20 years, many acting as individuals, though also many representing academic institutions and both user and vendor companies. Both the "Author" and "Committer" fields of such patches will reflect the committer. The actual author of a patch, if different, is generally listed in the commit message; reviewers or others who contributed ideas or otherwise helped with the patch may also be listed. Many patches, in the form in which they are committed, are the work of multiple people: original author or authors, reviewer(s), and/or committer. As a result, no simple analysis of duration or depth of contribution over time is possible from the commit log. The project operates a system of careful peer review and even committers have their work checked by other committers and the community as a whole. <br />
<br />
== New Committers and Removing Committers ==<br />
<br />
New committers are added approximately annually by the Core Team. Contributors to PostgreSQL are selected to be committers based on the following loose criteria:<br />
<br />
* several years of substantial contributions to the project<br />
* multiple and continuing code contributions<br />
* responsibility for maintenance of one or more areas of the codebase<br />
* track record of reviewing and helping other contributors with their patches<br />
* high quality code contributions which require very little revision or correction for commit<br />
* demonstrated understanding of the process and criteria for patch acceptance<br />
<br />
Generally, new committers are selected March or April and announced at pgCon.<br />
<br />
Committers who have become inactive and have not contributed significantly to the PostgreSQL project in several years are removed as committers. Again, the review process for this is approximately annual.<br />
<br />
[[Category:Community]]</div>Sfrost