https://wiki.postgresql.org/api.php?action=feedcontributions&feedformat=atom&user=Andresfreund 2026-06-09T15:57:08Z User contributions MediaWiki 1.39.17 https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42966 2026-03-06T16:02:58Z

<p>Andresfreund: </p> <hr /> <div>This page shows changes that I'd like to see in postgres, but that I am not personally working on at this point. Please feel free to work on any of these.<br /> <br /> == Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | System view showing what is holding back the xmin horizon<br /> | Today it takes a lot of knowledge to find all potential sources of holding back the xmin horizon. We should have a view showing all the sources of the horizon being held back.<br /> }}<br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Functions for getting block number / offset from a TID<br /> | I've way too many times written a regex to get block numbers out of tids. We should just have SQL level accessors.<br /> }}<br /> <br /> {{TodoItem<br /> | Lightweight memory contexts<br /> | For things like relcache entries, it'd be rather desirable to use fine-grained memory contexts, but using a full memory context would be rather heavyweight. We should create a new memory context type that does allocations using malloc() and tracks them in a linked list. That would obviously be a bit slower, but uses cases for it should never be heavy on allocations. We could also &quot;promote&quot; such contexts once the number of allocations increases.<br /> }}<br /> <br /> {{TodoItem<br /> | Make MemoryContext code usable in frontend code<br /> | Lots of frontend code would like to share code with backend code, including use of memory contexts. There are also complicated frontend binaries that could benefit from memory contexts.<br /> }}<br /> <br /> {{TodoItem<br /> | Split PANIC log level into failures that should dump core and ones that don't<br /> | Lots of reasons for PANIC do not benefit of a core file being generated. E.g. a PANIC due to ENOSPC during WAL file creation just makes the problem worse.<br /> }}<br /> <br /> {{TodoItem<br /> | Allow startup process to report errors that do not trigger a postmaster crash-restart cycle<br /> | There are errors detected by the startup process that will not be resolved by a crash-restart cycle, yet we react to them being thrown by doing a crash-restart cycle.<br /> }}<br /> <br /> {{TodoItem<br /> | Show time for index insertions in EXPLAIN ANALYZE of DML<br /> | Similar to how EXPLAIN ANALYZE shows the time for trigger invocations, it should have per-index instrumentation tracking how long the insertions into the different indexes take and how many buffer accesses were needed.<br /> }}<br /> <br /> {{TodoItem<br /> | Support EXPLAIN COPY <br /> | When bulk data loading it would be rather useful to see how long which trigger takes and how long index manipulations take (see prior TODO). When copying data out via COPY (query) TO ... it'd be useful to see the query plan.<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42060 2025-10-24T10:24:05Z

<p>Andresfreund: </p> <hr /> <div>This page shows changes that I'd like to see in postgres, but that I am not personally working on at this point. Please feel free to work on any of these.<br /> <br /> == Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | System view showing what is holding back the xmin horizon<br /> | Today it takes a lot of knowledge to find all potential sources of holding back the xmin horizon. We should have a view showing all the sources of the horizon being held back.<br /> }}<br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Functions for getting block number / offset from a TID<br /> | I've way too many times written a regex to get block numbers out of tids. We should just have SQL level accessors.<br /> }}<br /> <br /> {{TodoItem<br /> | Lightweight memory contexts<br /> | For things like relcache entries, it'd be rather desirable to use fine-grained memory contexts, but using a full memory context would be rather heavyweight. We should create a new memory context type that does allocations using malloc() and tracks them in a linked list. That would obviously be a bit slower, but uses cases for it should never be heavy on allocations. We could also &quot;promote&quot; such contexts once the number of allocations increases.<br /> }}<br /> <br /> {{TodoItem<br /> | Make MemoryContext code usable in frontend code<br /> | Lots of frontend code would like to share code with backend code, including use of memory contexts. There are also complicated frontend binaries that could benefit from memory contexts.<br /> }}<br /> <br /> {{TodoItem<br /> | Split PANIC log level into failures that should dump core and ones that don't<br /> | Lots of reasons for PANIC do not benefit of a core file being generated. E.g. a PANIC due to ENOSPC during WAL file creation just makes the problem worse.<br /> }}<br /> <br /> {{TodoItem<br /> | Allow startup process to report errors that do not trigger a postmaster crash-restart cycle<br /> | There are errors detected by the startup process that will not be resolved by a crash-restart cycle, yet we react to them being thrown by doing a crash-restart cycle.<br /> }}<br /> <br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42059 2025-10-24T10:14:33Z

<p>Andresfreund: </p> <hr /> <div>This page shows changes that I'd like to see in postgres, but that I am not personally working on at this point. Please feel free to work on any of these.<br /> <br /> == Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | System view showing what is holding back the xmin horizon<br /> | Today it takes a lot of knowledge to find all potential sources of holding back the xmin horizon. We should have a view showing all the sources of the horizon being held back.<br /> }}<br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Functions for getting block number / offset from a TID<br /> | I've way too many times written a regex to get block numbers out of tids. We should just have SQL level accessors.<br /> }}<br /> <br /> {{TodoItem<br /> | Lightweight memory contexts<br /> | For things like relcache entries, it'd be rather desirable to use fine-grained memory contexts, but using a full memory context would be rather heavyweight. We should create a new memory context type that does allocations using malloc() and tracks them in a linked list. That would obviously be a bit slower, but uses cases for it should never be heavy on allocations. We could also &quot;promote&quot; such contexts once the number of allocations increases.<br /> }}<br /> <br /> {{TodoItem<br /> | Make MemoryContext code usable in frontend code<br /> | Lots of frontend code would like to share code with backend code, including use of memory contexts. There are also complicated frontend binaries that could benefit from memory contexts.<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42058 2025-10-24T09:38:40Z

<p>Andresfreund: </p> <hr /> <div>This page shows changes that I'd like to see in postgres, but that I am not personally working on at this point. Please feel free to work on any of these.<br /> <br /> == Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | System view showing what is holding back the xmin horizon<br /> | Today it takes a lot of knowledge to find all potential sources of holding back the xmin horizon. We should have a view showing all the sources of the horizon being held back.<br /> }}<br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Functions for getting block number / offset from a TID<br /> | I've way too many times written a regex to get block numbers out of tids. We should just have SQL level accessors.<br /> }}<br /> <br /> {{TodoItem<br /> | Lightweight memory contexts<br /> | For things like relcache entries, it'd be rather desirable to use fine-grained memory contexts, but using a full memory context would be rather heavyweight. We should create a new memory context type that does allocations using malloc() and tracks them in a linked list. That would obviously be a bit slower, but uses cases for it should never be heavy on allocations. We could also &quot;promote&quot; such contexts once the number of allocations increases.<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42057 2025-10-24T09:28:50Z

<p>Andresfreund: </p> <hr /> <div>This page shows changes that I'd like to see in postgres, but that I am not personally working on at this point. Please feel free to work on any of these.<br /> <br /> == Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | System view showing what is holding back the xmin horizon<br /> | Today it takes a lot of knowledge to find all potential sources of holding back the xmin horizon. We should have a view showing all the sources of the horizon being held back.<br /> }}<br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Functions for getting block number / offset from a TID<br /> | I've way too many times written a regex to get block numbers out of tids. We should just have SQL level accessors.<br /> }}<br /> <br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42056 2025-10-24T09:24:57Z

<p>Andresfreund: </p> <hr /> <div>This page shows changes that I'd like to see in postgres, but that I am not personally working on at this point. Please feel free to work on any of these.<br /> <br /> == Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | System view showing what is holding back the xmin horizon<br /> | Today it takes a lot of knowledge to find all potential sources of holding back the xmin horizon. We should have a view showing all the sources of the horizon being held back.<br /> }}<br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42055 2025-10-24T09:23:54Z

<p>Andresfreund: </p> <hr /> <div>== Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | System view showing what is holding back the xmin horizon<br /> | Today it takes a lot of knowledge to find all potential sources of holding back the xmin horizon. We should have a view showing all the sources of the horizon being held back.<br /> }}<br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42054 2025-10-24T09:03:02Z

<p>Andresfreund: </p> <hr /> <div>== Stats ==<br /> <br /> {{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> <br /> == Benchmarking ==<br /> <br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> <br /> == Features ==<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42053 2025-10-24T09:00:58Z

<p>Andresfreund: </p> <hr /> <div>{{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> {{TodoItem<br /> | pgbench throttling with uniform distribution<br /> | The current Poisson distribution that pgbench -R uses adds so much noise that it's basically impossible to see changes in server latency<br /> }}<br /> <br /> {{TodoItem<br /> | pgbench setup steps as part of custom scripts<br /> | currently it's not possible to have distribute a single pgbench script that supports initializing the workload and running the benchmark as a single file<br /> }}<br /> <br /> {{TodoItem<br /> | Ensure that shared memory is populated at server start<br /> | Currently all shared memory is only actually allocated the first time memory is touched. That means that workloads have a prolonged warmup period until steady performance is reached.<br /> }}<br /> <br /> {{TodoItem<br /> | Add functionality to pre-allocate WAL segments<br /> | Performance of postgres changes wildly depending on whether WAL files are newly created or recycled (with the latter being much better on non-COW filesystems). We should provide a function to pre-create WAL segments.<br /> }}<br /> <br /> {{TodoItem<br /> | Add information about the number of newly created WAL segments to log_checkpoints output<br /> | While log_checkpoint output already includes information about added WAL segments, that information does not include WAL segments created by backends, making the number extremely misleading.<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}<br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42052 2025-10-24T07:51:19Z

<p>Andresfreund: </p> <hr /> <div>{{TodoItem<br /> | System View showing TCP statistics<br /> | This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility. While ss provides the necessary information, it makes it too hard to combine the information with postgres information (like which connections are for replication etc).<br /> }}<br /> <br /> {{TodoItem<br /> | System View showing OS memory usage<br /> | Currently it is very hard to know from within postgres which process actually uses how much memory. We should use operating system stats to show that information on operating systems where we know how.<br /> }}<br /> <br /> {{TodoItem<br /> | Introduce shared state for open relations<br /> | This is an enabling feature for many things:<br /> * AIO safe sync request queue (AIO needs to register dirty segments while in crit section)<br /> * safe caching of relation sizes, instead of constantly doing fstats<br /> * autovacuum partial trunctions without needing an AccessExclusiveLock (which causes query cancellations on standbys)<br /> * a more efficient buffer mapping would use a radix tree per relation, which in turn requires a place to &quot;hang off&quot; the radix tree <br /> }}<br /> <br /> <br /> {{TodoItem<br /> | Title<br /> | Desc<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund&diff=42051 2025-10-24T07:45:39Z

<p>Andresfreund: </p> <hr /> <div>List of changes to postgres that I'd like to see, but am not currently working on: [[User:Andresfreund/Desired Changes| Andres' desired changes]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund/Desired_Changes&diff=42050 2025-10-24T07:42:51Z

<p>Andresfreund: Created page with &quot;{{TodoItem |System View showing TCP statistics| This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility }}&quot;</p> <hr /> <div>{{TodoItem<br /> |System View showing TCP statistics| This would be useful to debug things like replication not keeping up, which is often caused by TCP settings. The statistics should show information as shown by the the [https://man7.org/linux/man-pages/man8/ss.8.html ss] utility<br /> }}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=User:Andresfreund&diff=42049 2025-10-24T07:39:26Z

<p>Andresfreund: Created page with &quot;List of changes to postgres that I&#039;d like to see, but am not currently working on:&quot;</p> <hr /> <div>List of changes to postgres that I'd like to see, but am not currently working on:</div>

Andresfreund https://wiki.postgresql.org/index.php?title=AIO&diff=41509 2025-07-28T20:20:25Z

<p>Andresfreund: </p> <hr /> <div>Effort to add asynchronous I/O, vectored/multiblock I/O and optional direct I/O to PostgreSQL.<br /> <br /> = High-level state =<br /> <br /> * Core AIO infrastructure merged, using workers and io_uring<br /> * Buffer manager infrastructure for asynchronous reads has been merged <br /> * Read streams use AIO for readahead, various places have been converted to use AIO (See [[#Users_of_read_streams|read stream users]])<br /> * Direct I/O can be enabled via the debug_io_direct=data GUC, but will be very slow for many workloads (See [[#Sub-Projects_required_for_.22minimal_real-world_use_of_DIO.22|DIO reqs]])<br /> <br /> = Sub-Projects required for &quot;minimal real-world use of DIO&quot; =<br /> <br /> * readahead of the table fetches for ordered index scans - without that common queries will be slower by orders of magnitude<br /> * Asynchronous writes for checkpointer / bgwriter and COPY<br /> * Some additional read streams might be required (e.g. ALTER TABLE SET TABLESPACE)<br /> <br /> = Streamification projects =<br /> <br /> In order to benefit from asynchronous I/O and vectored/multiblock I/O, we need to find all the places that pin buffers in a predictable way and teach them to do so via a stream. Switching to the streaming API insulates them from future improvements. For now, streamifying gets you vectored/multiblock I/O and systematic POSIX_FADV_WILLNEED hints only, but later true asynchronous modes will be proposed for the underlying infrastructure.<br /> <br /> == Users of read streams ==<br /> <br /> * [https://github.com/postgres/postgres/commit/b7b0f3f27241e424b7103397489464d907cef2c4 Use streaming I/O in sequential scans.] -- committed in v17<br /> * [https://github.com/postgres/postgres/commit/041b96802efa33d2bc9456f2ad946976b92b5ae1 Use streaming I/O in ANALYZE.] -- committed in 17<br /> * [https://github.com/postgres/postgres/commit/3a352df05e65de740b4a375a0ecbcae97a1f6196 Use streaming I/O in pg_prewarm.] -- committed in 17<br /> * [https://commitfest.postgresql.org/48/5089/ Use streaming I/O in VACUUM] -- work in progress<br /> * [https://commitfest.postgresql.org/48/4874/ Use streaming I/O in bitmap heap scans] -- work in progress<br /> * btree [https://www.postgresql.org/message-id/flat/cf85f46f-b02f-05b2-5248-5000b894ebab%40enterprisedb.com related thread] -- todo<br /> * brin -- todo<br /> * gist -- todo<br /> * gin -- todo<br /> * CREATE DATABASE STRATEGY=WAL_LOG<br /> * recovery -- todo<br /> * ...<br /> <br /> == Users of write streams ==<br /> <br /> There are not as many places that write data out.<br /> <br /> * CHECKPOINT -- work in progress<br /> * CREATE INDEX (buffer pool bypass, extending/replacing the bulk_write.c facility) -- work in progress<br /> * VACUUM writeback -- work in progress<br /> * COPY writeback -- work in progress<br /> * eviction writeback -- work in progress<br /> <br /> = Infrastructure changes =<br /> <br /> == Sub-Projects for AIO writes ==<br /> <br /> * bufmgr.c infrastructure to race-free acquire exclusive lock on buffer when IO might be in progress<br /> * sync.c requests cannot correctly be made in critical section, but that's required for buffered writes<br /> * Don't dirty pages while IO is ongoing (i.e. don't write hint bits while IO is ongoing)<br /> <br /> == Read stream API ==<br /> <br /> * basic buffered read stream, single relation + fork -- committed in 17<br /> * multi-relation read stream -- todo<br /> : Needed for recovery, but also useful for CREATE DATABASE ... STRATEGY=WAL_LOG.<br /> <br /> == Write stream API ==<br /> <br /> * basic buffered write stream -- todo<br /> * driving sync_file_range -- todo<br /> * driving WAL write frequency -- todo<br /> * buffer pool bypass, needed for CREATE INDEX (bulk writes) -- todo<br /> <br /> = TODO =<br /> <br /> == General ==<br /> <br /> * partition_prune failing in CI due to compiler bug in specific MSVC release<br /> ** [https://www.postgresql.org/message-id/CAApHDvof3AEVFZTT3MArh4Tq1cAuCJb_wk8u7FV5KMzJNhU4GQ%40mail.gmail.com discussion on -hackers]<br /> ** [https://github.com/cirruslabs/docker-images-windows/issues/23 ticket to update compiler on CI] <br /> ** drop commit &quot;XXX Add temporary workaround for partition_prune test on Windows&quot;<br /> * how should flush_range op be implemented on non-Linux?<br /> * write documentation, per-OS information<br /> * can we cut down on the number of places where we do non-blocking drain, for the benefit of implementations where that might be a system call?<br /> * replication is slow with wal_sync_method=open_datasync, because we don't call WalSndWakeupRequest(); this explains why eg the subscription tests are super slow on macOS on CI (macOS defaults to open_datasync)<br /> <br /> == Larger Issues ==<br /> <br /> * Streaming read / write interface are &quot;too local&quot; to specific users. A bit more backend global awareness would likely be a good idea<br /> * currently PostgreSQL does some prefetching with posix_fadvise_WILLNEED which is not asynchronous. See [[Readahead]].<br /> <br /> == io_method=worker ==<br /> <br /> * self-adjusting IO worker pool?<br /> * more work on the spurious-wakeup vs latency tradeoff<br /> <br /> == io_method=posix_aio ==<br /> <br /> * add an &quot;interruptible&quot; field in shmem that can be used to avoid useless wakeups while the submitter is running synchronous IOs or already draining, with some double-checked flags to avoid races?<br /> * detect presence of POSIX AIO automatically so you don't have to build --with-posix-aio<br /> ** would be good to pass smoke tests on all known POSIX AIO implementations before we do that; results so far:<br /> *** Successes!<br /> **** FreeBSD<br /> **** Linux with Glibc and Musl<br /> **** illumos and Solaris<br /> **** macOS<br /> ***** need to bump up sysctl limits to get decent performance<br /> **** AIX<br /> ***** needs shared_memory_type=sysv<br /> ***** aio_nwait() would be a better wait/reap interface than aio_suspend(), but sadly it can't wait for aio_fsync()<br /> ****** possible workaround: could use aio_nwait() whenever there are no aio_fsync calls outstanding<br /> *** Failures, that will probably need to be excluded by default in early versions:<br /> **** NetBSD 9 seems to spin eating 100% CPU in lio_listio() :-(<br /> ***** clue: an earlier iteration of this code was able to pass tests on NetBSD, when we were using aio_read(), aio_write() instead of lio_listio(), and when we were using SIGEV_SIGNAL instead of SIGEV_NONE<br /> ***** no bug report filed yet, but a NetBSD developer advised me not to try to use this, it's not ready<br /> ** if keeping it as a configure option, it should be &quot;enable/disable&quot;, not &quot;with/without&quot;<br /> * fixed<br /> ** [https://cirrus-ci.com/task/6024059377942528?logs=cores#L7 crash because aio_suspend() sees EINVAL], because &quot;activate&quot; LIO IOCB too soon, [https://github.com/anarazel/postgres/commit/b174a85c446d4190cc89847760427a031711d2a1 commit (fixup to be squashed)]<br /> ** DONE: can we avoid waiting for a merge head from a later generation? yes<br /> ** DONE: can we avoid using atomics in a signal handler? most uses removed but in flight count remains; could do somethign about this<br /> ** DONE: the naming and coding in the baton stuff is weird, needs a rethink -&gt; done, now called &quot;exchange&quot; and shared with iocp<br /> ** DONE: kill the array of active IOs if not using aio_suspend (eg FreeBSD)<br /> ** DONE: would it be better to have the signal handler give up after a short time so it can get back to doing something useful, and the waiter wake it again after a bit? -&gt; seems to be better, but may need some defences against thundering herds and useless wakeups<br /> <br /> == io_method=iocp ==<br /> <br /> * Acceptance criteria for moving iocp into the main aio branch (from aio-win32):<br /> ** &quot;TRAP: FailedAssertion(&quot;pgwin32_signal_event != NULL&quot;, File: &quot;C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\src\port\open.c&quot;, Line: 78, PID: 3416)&quot;<br /> *** pgwin32_open() currently hacked to comment that out because of unresolved ordering problem, read_nondefault_variables() vs pgwin32_signal_initialize()<br /> ** currently pgaio_can_scatter_gather() considers only io_data_direct when deciding; but it applies also to WAL I/O (and potentially, in future, who knows, temporary files etc)<br /> *** this matters primarily for Windows because Windows has a different answer depending on use of direct I/O, but we have at least 3 different GUCs to control that on diffrent subsystems<br /> *** one idea would be for pgaio_can_scatter_gather() to take an IO and check the &quot;scb&quot; to see who is asking, or something like that...<br /> *** another idea is to carry a &quot;direct&quot; flag on every IO so that the merge code doesn't have to concern itself with the details beyond that<br /> ** new API: pgaio_impl-&gt;opening_fd(int fd, int flags) so that Windows impl can register fd with IOCP if flags &amp; O_OVERLAPPED? currently that's all a bit kludgy<br /> * other things<br /> ** using GetQueuedCompletionEventEx() requires Windows Vista, but PostgreSQL currently targets XP+. both are long dead, but the case for bumping it needs to be made in the community <br /> ** does FileFlushBuffers() have an async cousin? doesn't look like it<br /> *** but there is an equivalen to fdatasync() in ntdll.dll; might be worth looking further<br /> ** pgaio_iocp_closing_fd() should drain only IOs on the given fd, not all IOs issued by this backend<br /> *** compare see pgaio_posix_aio_closing_fd() -- it only drains results, it does not reap, to avoid deadlock risk! need something like that here too<br /> <br /> * fixed <br /> ** DONE: calling it &quot;iocp&quot; for now (&quot;windows&quot; was too generic; we want to reserve the option to use the new Windows io_uring knockoff API which will probably be a separate method)<br /> ** DONE: kill the IOCP thread, and teach pgaio_windows_drain() to drain?<br /> ** DONE: solve the resulting deadlock by using the same procsignal trick as posix_aio?<br /> ** DONE: we should use GetQueuedCompletionEventEx() to consume multiple completions in one call, instead of a loop!<br /> <br /> == io_method=ioring ==<br /> <br /> * no code yet, just an idea<br /> * ioring (note: no 'u', maybe should be win_ioring or some other name) is a hypothetical future IO method that would use [https://windows-internals.com/i-o-rings-when-one-i-o-operation-is-not-enough/ Windows I/O rings], a knock-off of Linux io_uring that is available in Window 11 preview but still changing<br /> * so far the documentation only describes how to do reads but [https://twitter.com/axboe/status/1435614103314128897 we know that you can already do writes and flushes with the current preview], so there is enough there right now to write aio_ioring.c and hook it up<br /> * the easiest way could be to have one ioring per backend and use aio_exchange.c to deal with cross-process problems (like io_method=posix_aio and io_method=iocp), but it may also be possible to have N iorings that are somehow shared between processes and then use the context system for interlocking (like io_method=io_uring, the Linux one); can you do that, somehow share the handle + memory mapping for the s and c queues + correct wakeups?<br /> <br /> <br /> <br /> = Quick start for PostgreSQL hackers/reviewers =<br /> <br /> * Testing the default mode, simulated AIO using io_method=worker (= the default setting)<br /> ** Try strace-ing the backend and IO worker processes to see how I/O syscalls are offloaded<br /> ** Adjust the number of io_worker processes with io_workers=N<br /> ** See the view pg_stat_aios that shows individual IOs<br /> ** See the view pg_stat_aio_backends that shows per-backend info<br /> * Testing the use of direct I/O instead of PostgreSQL's traditional double buffering<br /> ** Set io_direct=data to disable OS buffering of relation data<br /> ** Set io_direct=wal to disable OS buffering WAL data<br /> ** Set io_direct=wal,wal_init,data to disable all<br /> * Testing OS-specific options for &quot;native&quot; AIO<br /> ** Linux io_uring<br /> *** install package liburing-dev (or liburing-devel on some distros)<br /> *** configure with --with-liburing (or if using Meson, it should find it by itself?)<br /> *** run with io_method=io_uring<br /> ** POSIX AIO, on macOS, FreeBSD, NetBSD, AIX, illumos, Solaris, Linux<br /> *** configure --with-posix-aio<br /> *** run with io_method=posix_aio<br /> *** Some OS specific notes:<br /> **** Linux POSIX AIO is fake, simulated by glibc and musl with threads, and works well enough for testing but wouldn't be a good choice to actually use<br /> **** Solaris/illumos also has fake POSIX AIO (there is a more native/kernel supported AIO API but we don't support that)<br /> **** macOS has very tight limits on AIO; like every other application that uses AIO we're going to have to publish recommentations to crank them up ([http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.dc39123_1254/html/qinstmac/qinstmac11.htm Sybase for example], VirtualBox is another); on the bright side, macOS defaults provide a nice workout for the IO retry code paths; that is, often we try to start an IO and the kernel says EAGAIN, so we way for one IO to complete and then try again, and you can see this in the pg_stat_aio_backends retry counter column<br /> **** AIX only works if you set shared_memory_type=sysv because AIX's AIO can't access memory we allocate with mmap() (otherwise all IO fails with EFAULT)<br /> **** AIX direct I/O might be unnecessarily serializing per-file, which we could fix with O_CONCURRENT, O_CIO or O_CIOR<br /> **** HPUX probably also needs O_CIO to avoid serializing direct IO (native AIO not working there yet but this applies to worker mode too)<br /> ** Windows IOCP<br /> *** run with io_method=iocp<br /> *** (not yet pushed to main aio branch, find it in the aio-win32 branch until it's a little more complete...)</div>

Andresfreund https://wiki.postgresql.org/index.php?title=AIO&diff=41508 2025-07-28T18:30:50Z

<p>Andresfreund: /* Sub-Projects required for &quot;minimal real-world use of DIO&quot; */</p> <hr /> <div>Effort to add asynchronous I/O, vectored/multiblock I/O and optional direct I/O to PostgreSQL.<br /> <br /> = High-level state =<br /> <br /> * Core AIO infrastructure merged, using workers and io_uring<br /> * Buffer manager infrastructure for asynchronous reads has been merged <br /> * Read streams use AIO for readahead, various places have been converted to use AIO [[#Users_of_read_streams]]<br /> * Direct I/O can be enabled via the debug_io_direct=data GUC, but will be very slow for many workloads [[#Sub-Projects_required_for_.22minimal_real-world_use_of_DIO.22]]<br /> <br /> = Sub-Projects required for &quot;minimal real-world use of DIO&quot; =<br /> <br /> * readahead of the table fetches for ordered index scans - without that common queries will be slower by orders of magnitude<br /> * Asynchronous writes for checkpointer / bgwriter and COPY<br /> * Some additional read streams might be required (e.g. ALTER TABLE SET TABLESPACE)<br /> <br /> = Sub-Projects for AIO writes = <br /> <br /> * bufmgr.c infrastructure to race-free acquire exclusive lock on buffer when IO might be in progress<br /> * sync.c requests cannot correctly be made in critical section, but that's required for buffered writes<br /> * Don't dirty pages while IO is ongoing (i.e. don't write hint bits while IO is ongoing)<br /> <br /> = Streamification projects =<br /> <br /> In order to benefit from asynchronous I/O and vectored/multiblock I/O, we need to find all the places that pin buffers in a predictable way and teach them to do so via a stream. Switching to the streaming API insulates them from future improvements. For now, streamifying gets you vectored/multiblock I/O and systematic POSIX_FADV_WILLNEED hints only, but later true asynchronous modes will be proposed for the underlying infrastructure.<br /> <br /> == Users of read streams ==<br /> <br /> * [https://github.com/postgres/postgres/commit/b7b0f3f27241e424b7103397489464d907cef2c4 Use streaming I/O in sequential scans.] -- committed in v17<br /> * [https://github.com/postgres/postgres/commit/041b96802efa33d2bc9456f2ad946976b92b5ae1 Use streaming I/O in ANALYZE.] -- committed in 17<br /> * [https://github.com/postgres/postgres/commit/3a352df05e65de740b4a375a0ecbcae97a1f6196 Use streaming I/O in pg_prewarm.] -- committed in 17<br /> * [https://commitfest.postgresql.org/48/5089/ Use streaming I/O in VACUUM] -- work in progress<br /> * [https://commitfest.postgresql.org/48/4874/ Use streaming I/O in bitmap heap scans] -- work in progress<br /> * btree [https://www.postgresql.org/message-id/flat/cf85f46f-b02f-05b2-5248-5000b894ebab%40enterprisedb.com related thread] -- todo<br /> * brin -- todo<br /> * gist -- todo<br /> * gin -- todo<br /> * CREATE DATABASE STRATEGY=WAL_LOG<br /> * recovery -- todo<br /> * ...<br /> <br /> == Users of write streams ==<br /> <br /> There are not as many places that write data out.<br /> <br /> * CHECKPOINT -- work in progress<br /> * CREATE INDEX (buffer pool bypass, extending/replacing the bulk_write.c facility) -- work in progress<br /> * VACUUM writeback -- work in progress<br /> * COPY writeback -- work in progress<br /> * eviction writeback -- work in progress<br /> <br /> = Infrastructure changes =<br /> <br /> == Read stream API ==<br /> <br /> * basic buffered read stream, single relation + fork -- committed in 17<br /> * multi-relation read stream -- todo<br /> : Needed for recovery, but also useful for CREATE DATABASE ... STRATEGY=WAL_LOG.<br /> <br /> == Write stream API ==<br /> <br /> * basic buffered write stream -- todo<br /> * driving sync_file_range -- todo<br /> * driving WAL write frequency -- todo<br /> * buffer pool bypass, needed for CREATE INDEX (bulk writes) -- todo<br /> <br /> == Committed ==<br /> <br /> * [https://commitfest.postgresql.org/33/3147/ O_DIRECT on macOS]<br /> ** So that Mac hackers can test io_data_direct=on (worker mode or posix_aio mode)<br /> * [https://commitfest.postgresql.org/31/2885/ pg_pwritev() and pg_preadv()]<br /> ** Portable support for synchronous scatter/gather I/O<br /> * [https://commitfest.postgresql.org/32/3008/ Replace buffer I/O locks with condition variables]<br /> ** So that backends other that the one that drains an I/O from the kernel can wait for it to complete<br /> * [https://commitfest.postgresql.org/41/3999/ Aligned memory allocation]<br /> ** As required for buffers used in direct I/O<br /> * [https://commitfest.postgresql.org/42/3996/ Direct I/O GUC]<br /> ** Released in 16 as debug_io_direct, to avoid attracting too much attention<br /> * [https://commitfest.postgresql.org/34/2962/ Fix DROP TABLESPACE on Windows with ProcSignalBarrier]<br /> ** This is actually an ancient bug in PostgreSQL on Windows, but it was a bit harder to hit; make check fails every time under io_method=worker because processes hang around holding cached fds.<br /> * [https://commitfest.postgresql.org/42/3993/ Refactor relation extension]<br /> ** This includes work to allow a backend to have multiple buffers in BM_IO_IN_PROGRESS state.<br /> * [https://commitfest.postgresql.org/44/4532/ Streaming I/O, vectored I/O]<br /> ** Introduce the concept of streams of buffers, initially for more efficient synchronous I/O<br /> <br /> = TODO =<br /> <br /> == General ==<br /> <br /> * partition_prune failing in CI due to compiler bug in specific MSVC release<br /> ** [https://www.postgresql.org/message-id/CAApHDvof3AEVFZTT3MArh4Tq1cAuCJb_wk8u7FV5KMzJNhU4GQ%40mail.gmail.com discussion on -hackers]<br /> ** [https://github.com/cirruslabs/docker-images-windows/issues/23 ticket to update compiler on CI] <br /> ** drop commit &quot;XXX Add temporary workaround for partition_prune test on Windows&quot;<br /> * how should flush_range op be implemented on non-Linux?<br /> * write documentation, per-OS information<br /> * can we cut down on the number of places where we do non-blocking drain, for the benefit of implementations where that might be a system call?<br /> * replication is slow with wal_sync_method=open_datasync, because we don't call WalSndWakeupRequest(); this explains why eg the subscription tests are super slow on macOS on CI (macOS defaults to open_datasync)<br /> <br /> == Larger Issues ==<br /> <br /> * Streaming read / write interface are &quot;too local&quot; to specific users. A bit more backend global awareness would likely be a good idea<br /> * currently PostgreSQL does some prefetching with posix_fadvise_WILLNEED which is not asynchronous. See [[Readahead]].<br /> <br /> == io_method=worker ==<br /> <br /> * self-adjusting IO worker pool?<br /> * more work on the spurious-wakeup vs latency tradeoff<br /> <br /> == io_method=posix_aio ==<br /> <br /> * add an &quot;interruptible&quot; field in shmem that can be used to avoid useless wakeups while the submitter is running synchronous IOs or already draining, with some double-checked flags to avoid races?<br /> * detect presence of POSIX AIO automatically so you don't have to build --with-posix-aio<br /> ** would be good to pass smoke tests on all known POSIX AIO implementations before we do that; results so far:<br /> *** Successes!<br /> **** FreeBSD<br /> **** Linux with Glibc and Musl<br /> **** illumos and Solaris<br /> **** macOS<br /> ***** need to bump up sysctl limits to get decent performance<br /> **** AIX<br /> ***** needs shared_memory_type=sysv<br /> ***** aio_nwait() would be a better wait/reap interface than aio_suspend(), but sadly it can't wait for aio_fsync()<br /> ****** possible workaround: could use aio_nwait() whenever there are no aio_fsync calls outstanding<br /> *** Failures, that will probably need to be excluded by default in early versions:<br /> **** NetBSD 9 seems to spin eating 100% CPU in lio_listio() :-(<br /> ***** clue: an earlier iteration of this code was able to pass tests on NetBSD, when we were using aio_read(), aio_write() instead of lio_listio(), and when we were using SIGEV_SIGNAL instead of SIGEV_NONE<br /> ***** no bug report filed yet, but a NetBSD developer advised me not to try to use this, it's not ready<br /> ** if keeping it as a configure option, it should be &quot;enable/disable&quot;, not &quot;with/without&quot;<br /> * fixed<br /> ** [https://cirrus-ci.com/task/6024059377942528?logs=cores#L7 crash because aio_suspend() sees EINVAL], because &quot;activate&quot; LIO IOCB too soon, [https://github.com/anarazel/postgres/commit/b174a85c446d4190cc89847760427a031711d2a1 commit (fixup to be squashed)]<br /> ** DONE: can we avoid waiting for a merge head from a later generation? yes<br /> ** DONE: can we avoid using atomics in a signal handler? most uses removed but in flight count remains; could do somethign about this<br /> ** DONE: the naming and coding in the baton stuff is weird, needs a rethink -&gt; done, now called &quot;exchange&quot; and shared with iocp<br /> ** DONE: kill the array of active IOs if not using aio_suspend (eg FreeBSD)<br /> ** DONE: would it be better to have the signal handler give up after a short time so it can get back to doing something useful, and the waiter wake it again after a bit? -&gt; seems to be better, but may need some defences against thundering herds and useless wakeups<br /> <br /> == io_method=iocp ==<br /> <br /> * Acceptance criteria for moving iocp into the main aio branch (from aio-win32):<br /> ** &quot;TRAP: FailedAssertion(&quot;pgwin32_signal_event != NULL&quot;, File: &quot;C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\src\port\open.c&quot;, Line: 78, PID: 3416)&quot;<br /> *** pgwin32_open() currently hacked to comment that out because of unresolved ordering problem, read_nondefault_variables() vs pgwin32_signal_initialize()<br /> ** currently pgaio_can_scatter_gather() considers only io_data_direct when deciding; but it applies also to WAL I/O (and potentially, in future, who knows, temporary files etc)<br /> *** this matters primarily for Windows because Windows has a different answer depending on use of direct I/O, but we have at least 3 different GUCs to control that on diffrent subsystems<br /> *** one idea would be for pgaio_can_scatter_gather() to take an IO and check the &quot;scb&quot; to see who is asking, or something like that...<br /> *** another idea is to carry a &quot;direct&quot; flag on every IO so that the merge code doesn't have to concern itself with the details beyond that<br /> ** new API: pgaio_impl-&gt;opening_fd(int fd, int flags) so that Windows impl can register fd with IOCP if flags &amp; O_OVERLAPPED? currently that's all a bit kludgy<br /> * other things<br /> ** using GetQueuedCompletionEventEx() requires Windows Vista, but PostgreSQL currently targets XP+. both are long dead, but the case for bumping it needs to be made in the community <br /> ** does FileFlushBuffers() have an async cousin? doesn't look like it<br /> *** but there is an equivalen to fdatasync() in ntdll.dll; might be worth looking further<br /> ** pgaio_iocp_closing_fd() should drain only IOs on the given fd, not all IOs issued by this backend<br /> *** compare see pgaio_posix_aio_closing_fd() -- it only drains results, it does not reap, to avoid deadlock risk! need something like that here too<br /> <br /> * fixed <br /> ** DONE: calling it &quot;iocp&quot; for now (&quot;windows&quot; was too generic; we want to reserve the option to use the new Windows io_uring knockoff API which will probably be a separate method)<br /> ** DONE: kill the IOCP thread, and teach pgaio_windows_drain() to drain?<br /> ** DONE: solve the resulting deadlock by using the same procsignal trick as posix_aio?<br /> ** DONE: we should use GetQueuedCompletionEventEx() to consume multiple completions in one call, instead of a loop!<br /> <br /> == io_method=ioring ==<br /> <br /> * no code yet, just an idea<br /> * ioring (note: no 'u', maybe should be win_ioring or some other name) is a hypothetical future IO method that would use [https://windows-internals.com/i-o-rings-when-one-i-o-operation-is-not-enough/ Windows I/O rings], a knock-off of Linux io_uring that is available in Window 11 preview but still changing<br /> * so far the documentation only describes how to do reads but [https://twitter.com/axboe/status/1435614103314128897 we know that you can already do writes and flushes with the current preview], so there is enough there right now to write aio_ioring.c and hook it up<br /> * the easiest way could be to have one ioring per backend and use aio_exchange.c to deal with cross-process problems (like io_method=posix_aio and io_method=iocp), but it may also be possible to have N iorings that are somehow shared between processes and then use the context system for interlocking (like io_method=io_uring, the Linux one); can you do that, somehow share the handle + memory mapping for the s and c queues + correct wakeups?<br /> <br /> <br /> <br /> = Quick start for PostgreSQL hackers/reviewers =<br /> <br /> * Testing the default mode, simulated AIO using io_method=worker (= the default setting)<br /> ** Try strace-ing the backend and IO worker processes to see how I/O syscalls are offloaded<br /> ** Adjust the number of io_worker processes with io_workers=N<br /> ** See the view pg_stat_aios that shows individual IOs<br /> ** See the view pg_stat_aio_backends that shows per-backend info<br /> * Testing the use of direct I/O instead of PostgreSQL's traditional double buffering<br /> ** Set io_direct=data to disable OS buffering of relation data<br /> ** Set io_direct=wal to disable OS buffering WAL data<br /> ** Set io_direct=wal,wal_init,data to disable all<br /> * Testing OS-specific options for &quot;native&quot; AIO<br /> ** Linux io_uring<br /> *** install package liburing-dev (or liburing-devel on some distros)<br /> *** configure with --with-liburing (or if using Meson, it should find it by itself?)<br /> *** run with io_method=io_uring<br /> ** POSIX AIO, on macOS, FreeBSD, NetBSD, AIX, illumos, Solaris, Linux<br /> *** configure --with-posix-aio<br /> *** run with io_method=posix_aio<br /> *** Some OS specific notes:<br /> **** Linux POSIX AIO is fake, simulated by glibc and musl with threads, and works well enough for testing but wouldn't be a good choice to actually use<br /> **** Solaris/illumos also has fake POSIX AIO (there is a more native/kernel supported AIO API but we don't support that)<br /> **** macOS has very tight limits on AIO; like every other application that uses AIO we're going to have to publish recommentations to crank them up ([http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.dc39123_1254/html/qinstmac/qinstmac11.htm Sybase for example], VirtualBox is another); on the bright side, macOS defaults provide a nice workout for the IO retry code paths; that is, often we try to start an IO and the kernel says EAGAIN, so we way for one IO to complete and then try again, and you can see this in the pg_stat_aio_backends retry counter column<br /> **** AIX only works if you set shared_memory_type=sysv because AIX's AIO can't access memory we allocate with mmap() (otherwise all IO fails with EFAULT)<br /> **** AIX direct I/O might be unnecessarily serializing per-file, which we could fix with O_CONCURRENT, O_CIO or O_CIOR<br /> **** HPUX probably also needs O_CIO to avoid serializing direct IO (native AIO not working there yet but this applies to worker mode too)<br /> ** Windows IOCP<br /> *** run with io_method=iocp<br /> *** (not yet pushed to main aio branch, find it in the aio-win32 branch until it's a little more complete...)</div>

Andresfreund https://wiki.postgresql.org/index.php?title=AIO&diff=41507 2025-07-28T18:26:26Z

<p>Andresfreund: </p> <hr /> <div>Effort to add asynchronous I/O, vectored/multiblock I/O and optional direct I/O to PostgreSQL.<br /> <br /> = High-level state =<br /> <br /> * Core AIO infrastructure merged, using workers and io_uring<br /> * Buffer manager infrastructure for asynchronous reads has been merged <br /> * Read streams use AIO for readahead, various places have been converted to use AIO [[#Users_of_read_streams]]<br /> * Direct I/O can be enabled via the debug_io_direct=data GUC, but will be very slow for many workloads [[#Sub-Projects_required_for_.22minimal_real-world_use_of_DIO.22]]<br /> <br /> = Sub-Projects required for &quot;minimal real-world use of DIO&quot; =<br /> <br /> * readahead of the table fetches for ordered index scans<br /> * Asynchronous writes for checkpointer / bgwriter and COPY<br /> * Some additional read streams might be required (e.g. ALTER TABLE SET TABLESPACE)<br /> <br /> = Sub-Projects for AIO writes = <br /> <br /> * bufmgr.c infrastructure to race-free acquire exclusive lock on buffer when IO might be in progress<br /> * sync.c requests cannot correctly be made in critical section, but that's required for buffered writes<br /> * Don't dirty pages while IO is ongoing (i.e. don't write hint bits while IO is ongoing)<br /> <br /> = Streamification projects =<br /> <br /> In order to benefit from asynchronous I/O and vectored/multiblock I/O, we need to find all the places that pin buffers in a predictable way and teach them to do so via a stream. Switching to the streaming API insulates them from future improvements. For now, streamifying gets you vectored/multiblock I/O and systematic POSIX_FADV_WILLNEED hints only, but later true asynchronous modes will be proposed for the underlying infrastructure.<br /> <br /> == Users of read streams ==<br /> <br /> * [https://github.com/postgres/postgres/commit/b7b0f3f27241e424b7103397489464d907cef2c4 Use streaming I/O in sequential scans.] -- committed in v17<br /> * [https://github.com/postgres/postgres/commit/041b96802efa33d2bc9456f2ad946976b92b5ae1 Use streaming I/O in ANALYZE.] -- committed in 17<br /> * [https://github.com/postgres/postgres/commit/3a352df05e65de740b4a375a0ecbcae97a1f6196 Use streaming I/O in pg_prewarm.] -- committed in 17<br /> * [https://commitfest.postgresql.org/48/5089/ Use streaming I/O in VACUUM] -- work in progress<br /> * [https://commitfest.postgresql.org/48/4874/ Use streaming I/O in bitmap heap scans] -- work in progress<br /> * btree [https://www.postgresql.org/message-id/flat/cf85f46f-b02f-05b2-5248-5000b894ebab%40enterprisedb.com related thread] -- todo<br /> * brin -- todo<br /> * gist -- todo<br /> * gin -- todo<br /> * CREATE DATABASE STRATEGY=WAL_LOG<br /> * recovery -- todo<br /> * ...<br /> <br /> == Users of write streams ==<br /> <br /> There are not as many places that write data out.<br /> <br /> * CHECKPOINT -- work in progress<br /> * CREATE INDEX (buffer pool bypass, extending/replacing the bulk_write.c facility) -- work in progress<br /> * VACUUM writeback -- work in progress<br /> * COPY writeback -- work in progress<br /> * eviction writeback -- work in progress<br /> <br /> = Infrastructure changes =<br /> <br /> == Read stream API ==<br /> <br /> * basic buffered read stream, single relation + fork -- committed in 17<br /> * multi-relation read stream -- todo<br /> : Needed for recovery, but also useful for CREATE DATABASE ... STRATEGY=WAL_LOG.<br /> <br /> == Write stream API ==<br /> <br /> * basic buffered write stream -- todo<br /> * driving sync_file_range -- todo<br /> * driving WAL write frequency -- todo<br /> * buffer pool bypass, needed for CREATE INDEX (bulk writes) -- todo<br /> <br /> == Committed ==<br /> <br /> * [https://commitfest.postgresql.org/33/3147/ O_DIRECT on macOS]<br /> ** So that Mac hackers can test io_data_direct=on (worker mode or posix_aio mode)<br /> * [https://commitfest.postgresql.org/31/2885/ pg_pwritev() and pg_preadv()]<br /> ** Portable support for synchronous scatter/gather I/O<br /> * [https://commitfest.postgresql.org/32/3008/ Replace buffer I/O locks with condition variables]<br /> ** So that backends other that the one that drains an I/O from the kernel can wait for it to complete<br /> * [https://commitfest.postgresql.org/41/3999/ Aligned memory allocation]<br /> ** As required for buffers used in direct I/O<br /> * [https://commitfest.postgresql.org/42/3996/ Direct I/O GUC]<br /> ** Released in 16 as debug_io_direct, to avoid attracting too much attention<br /> * [https://commitfest.postgresql.org/34/2962/ Fix DROP TABLESPACE on Windows with ProcSignalBarrier]<br /> ** This is actually an ancient bug in PostgreSQL on Windows, but it was a bit harder to hit; make check fails every time under io_method=worker because processes hang around holding cached fds.<br /> * [https://commitfest.postgresql.org/42/3993/ Refactor relation extension]<br /> ** This includes work to allow a backend to have multiple buffers in BM_IO_IN_PROGRESS state.<br /> * [https://commitfest.postgresql.org/44/4532/ Streaming I/O, vectored I/O]<br /> ** Introduce the concept of streams of buffers, initially for more efficient synchronous I/O<br /> <br /> = TODO =<br /> <br /> == General ==<br /> <br /> * partition_prune failing in CI due to compiler bug in specific MSVC release<br /> ** [https://www.postgresql.org/message-id/CAApHDvof3AEVFZTT3MArh4Tq1cAuCJb_wk8u7FV5KMzJNhU4GQ%40mail.gmail.com discussion on -hackers]<br /> ** [https://github.com/cirruslabs/docker-images-windows/issues/23 ticket to update compiler on CI] <br /> ** drop commit &quot;XXX Add temporary workaround for partition_prune test on Windows&quot;<br /> * how should flush_range op be implemented on non-Linux?<br /> * write documentation, per-OS information<br /> * can we cut down on the number of places where we do non-blocking drain, for the benefit of implementations where that might be a system call?<br /> * replication is slow with wal_sync_method=open_datasync, because we don't call WalSndWakeupRequest(); this explains why eg the subscription tests are super slow on macOS on CI (macOS defaults to open_datasync)<br /> <br /> == Larger Issues ==<br /> <br /> * Streaming read / write interface are &quot;too local&quot; to specific users. A bit more backend global awareness would likely be a good idea<br /> * currently PostgreSQL does some prefetching with posix_fadvise_WILLNEED which is not asynchronous. See [[Readahead]].<br /> <br /> == io_method=worker ==<br /> <br /> * self-adjusting IO worker pool?<br /> * more work on the spurious-wakeup vs latency tradeoff<br /> <br /> == io_method=posix_aio ==<br /> <br /> * add an &quot;interruptible&quot; field in shmem that can be used to avoid useless wakeups while the submitter is running synchronous IOs or already draining, with some double-checked flags to avoid races?<br /> * detect presence of POSIX AIO automatically so you don't have to build --with-posix-aio<br /> ** would be good to pass smoke tests on all known POSIX AIO implementations before we do that; results so far:<br /> *** Successes!<br /> **** FreeBSD<br /> **** Linux with Glibc and Musl<br /> **** illumos and Solaris<br /> **** macOS<br /> ***** need to bump up sysctl limits to get decent performance<br /> **** AIX<br /> ***** needs shared_memory_type=sysv<br /> ***** aio_nwait() would be a better wait/reap interface than aio_suspend(), but sadly it can't wait for aio_fsync()<br /> ****** possible workaround: could use aio_nwait() whenever there are no aio_fsync calls outstanding<br /> *** Failures, that will probably need to be excluded by default in early versions:<br /> **** NetBSD 9 seems to spin eating 100% CPU in lio_listio() :-(<br /> ***** clue: an earlier iteration of this code was able to pass tests on NetBSD, when we were using aio_read(), aio_write() instead of lio_listio(), and when we were using SIGEV_SIGNAL instead of SIGEV_NONE<br /> ***** no bug report filed yet, but a NetBSD developer advised me not to try to use this, it's not ready<br /> ** if keeping it as a configure option, it should be &quot;enable/disable&quot;, not &quot;with/without&quot;<br /> * fixed<br /> ** [https://cirrus-ci.com/task/6024059377942528?logs=cores#L7 crash because aio_suspend() sees EINVAL], because &quot;activate&quot; LIO IOCB too soon, [https://github.com/anarazel/postgres/commit/b174a85c446d4190cc89847760427a031711d2a1 commit (fixup to be squashed)]<br /> ** DONE: can we avoid waiting for a merge head from a later generation? yes<br /> ** DONE: can we avoid using atomics in a signal handler? most uses removed but in flight count remains; could do somethign about this<br /> ** DONE: the naming and coding in the baton stuff is weird, needs a rethink -&gt; done, now called &quot;exchange&quot; and shared with iocp<br /> ** DONE: kill the array of active IOs if not using aio_suspend (eg FreeBSD)<br /> ** DONE: would it be better to have the signal handler give up after a short time so it can get back to doing something useful, and the waiter wake it again after a bit? -&gt; seems to be better, but may need some defences against thundering herds and useless wakeups<br /> <br /> == io_method=iocp ==<br /> <br /> * Acceptance criteria for moving iocp into the main aio branch (from aio-win32):<br /> ** &quot;TRAP: FailedAssertion(&quot;pgwin32_signal_event != NULL&quot;, File: &quot;C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\src\port\open.c&quot;, Line: 78, PID: 3416)&quot;<br /> *** pgwin32_open() currently hacked to comment that out because of unresolved ordering problem, read_nondefault_variables() vs pgwin32_signal_initialize()<br /> ** currently pgaio_can_scatter_gather() considers only io_data_direct when deciding; but it applies also to WAL I/O (and potentially, in future, who knows, temporary files etc)<br /> *** this matters primarily for Windows because Windows has a different answer depending on use of direct I/O, but we have at least 3 different GUCs to control that on diffrent subsystems<br /> *** one idea would be for pgaio_can_scatter_gather() to take an IO and check the &quot;scb&quot; to see who is asking, or something like that...<br /> *** another idea is to carry a &quot;direct&quot; flag on every IO so that the merge code doesn't have to concern itself with the details beyond that<br /> ** new API: pgaio_impl-&gt;opening_fd(int fd, int flags) so that Windows impl can register fd with IOCP if flags &amp; O_OVERLAPPED? currently that's all a bit kludgy<br /> * other things<br /> ** using GetQueuedCompletionEventEx() requires Windows Vista, but PostgreSQL currently targets XP+. both are long dead, but the case for bumping it needs to be made in the community <br /> ** does FileFlushBuffers() have an async cousin? doesn't look like it<br /> *** but there is an equivalen to fdatasync() in ntdll.dll; might be worth looking further<br /> ** pgaio_iocp_closing_fd() should drain only IOs on the given fd, not all IOs issued by this backend<br /> *** compare see pgaio_posix_aio_closing_fd() -- it only drains results, it does not reap, to avoid deadlock risk! need something like that here too<br /> <br /> * fixed <br /> ** DONE: calling it &quot;iocp&quot; for now (&quot;windows&quot; was too generic; we want to reserve the option to use the new Windows io_uring knockoff API which will probably be a separate method)<br /> ** DONE: kill the IOCP thread, and teach pgaio_windows_drain() to drain?<br /> ** DONE: solve the resulting deadlock by using the same procsignal trick as posix_aio?<br /> ** DONE: we should use GetQueuedCompletionEventEx() to consume multiple completions in one call, instead of a loop!<br /> <br /> == io_method=ioring ==<br /> <br /> * no code yet, just an idea<br /> * ioring (note: no 'u', maybe should be win_ioring or some other name) is a hypothetical future IO method that would use [https://windows-internals.com/i-o-rings-when-one-i-o-operation-is-not-enough/ Windows I/O rings], a knock-off of Linux io_uring that is available in Window 11 preview but still changing<br /> * so far the documentation only describes how to do reads but [https://twitter.com/axboe/status/1435614103314128897 we know that you can already do writes and flushes with the current preview], so there is enough there right now to write aio_ioring.c and hook it up<br /> * the easiest way could be to have one ioring per backend and use aio_exchange.c to deal with cross-process problems (like io_method=posix_aio and io_method=iocp), but it may also be possible to have N iorings that are somehow shared between processes and then use the context system for interlocking (like io_method=io_uring, the Linux one); can you do that, somehow share the handle + memory mapping for the s and c queues + correct wakeups?<br /> <br /> <br /> <br /> = Quick start for PostgreSQL hackers/reviewers =<br /> <br /> * Testing the default mode, simulated AIO using io_method=worker (= the default setting)<br /> ** Try strace-ing the backend and IO worker processes to see how I/O syscalls are offloaded<br /> ** Adjust the number of io_worker processes with io_workers=N<br /> ** See the view pg_stat_aios that shows individual IOs<br /> ** See the view pg_stat_aio_backends that shows per-backend info<br /> * Testing the use of direct I/O instead of PostgreSQL's traditional double buffering<br /> ** Set io_direct=data to disable OS buffering of relation data<br /> ** Set io_direct=wal to disable OS buffering WAL data<br /> ** Set io_direct=wal,wal_init,data to disable all<br /> * Testing OS-specific options for &quot;native&quot; AIO<br /> ** Linux io_uring<br /> *** install package liburing-dev (or liburing-devel on some distros)<br /> *** configure with --with-liburing (or if using Meson, it should find it by itself?)<br /> *** run with io_method=io_uring<br /> ** POSIX AIO, on macOS, FreeBSD, NetBSD, AIX, illumos, Solaris, Linux<br /> *** configure --with-posix-aio<br /> *** run with io_method=posix_aio<br /> *** Some OS specific notes:<br /> **** Linux POSIX AIO is fake, simulated by glibc and musl with threads, and works well enough for testing but wouldn't be a good choice to actually use<br /> **** Solaris/illumos also has fake POSIX AIO (there is a more native/kernel supported AIO API but we don't support that)<br /> **** macOS has very tight limits on AIO; like every other application that uses AIO we're going to have to publish recommentations to crank them up ([http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.dc39123_1254/html/qinstmac/qinstmac11.htm Sybase for example], VirtualBox is another); on the bright side, macOS defaults provide a nice workout for the IO retry code paths; that is, often we try to start an IO and the kernel says EAGAIN, so we way for one IO to complete and then try again, and you can see this in the pg_stat_aio_backends retry counter column<br /> **** AIX only works if you set shared_memory_type=sysv because AIX's AIO can't access memory we allocate with mmap() (otherwise all IO fails with EFAULT)<br /> **** AIX direct I/O might be unnecessarily serializing per-file, which we could fix with O_CONCURRENT, O_CIO or O_CIOR<br /> **** HPUX probably also needs O_CIO to avoid serializing direct IO (native AIO not working there yet but this applies to worker mode too)<br /> ** Windows IOCP<br /> *** run with io_method=iocp<br /> *** (not yet pushed to main aio branch, find it in the aio-win32 branch until it's a little more complete...)</div>

Andresfreund https://wiki.postgresql.org/index.php?title=AIO&diff=41506 2025-07-28T17:49:05Z

<p>Andresfreund: </p> <hr /> <div>Effort to add asynchronous I/O, vectored/multiblock I/O and optional direct I/O to PostgreSQL.<br /> <br /> = High-level state =<br /> <br /> * Core AIO infrastructure merged, using workers and io_uring<br /> * Buffer manager infrastructure for asynchronous reads has been merged <br /> * Read streams use AIO for readahead, various places have been converted to use AIO [[#Users_of_read_streams]]<br /> <br /> = Sub-Projects required for &quot;minimal real-world use of DIO&quot; =<br /> <br /> * readahead of the table fetches for ordered index scans<br /> * Asynchronous writes for checkpointer / bgwriter and COPY<br /> * Some additional read streams might be required (e.g. ALTER TABLE SET TABLESPACE)<br /> <br /> = Sub-Projects for AIO writes = <br /> <br /> * bufmgr.c infrastructure to race-free acquire exclusive lock on buffer when IO might be in progress<br /> * sync.c requests cannot correctly be made in critical section, but that's required for buffered writes<br /> * Don't dirty pages while IO is ongoing (i.e. don't write hint bits while IO is ongoing)<br /> <br /> = Streamification projects =<br /> <br /> In order to benefit from asynchronous I/O and vectored/multiblock I/O, we need to find all the places that pin buffers in a predictable way and teach them to do so via a stream. Switching to the streaming API insulates them from future improvements. For now, streamifying gets you vectored/multiblock I/O and systematic POSIX_FADV_WILLNEED hints only, but later true asynchronous modes will be proposed for the underlying infrastructure.<br /> <br /> == Users of read streams ==<br /> <br /> * [https://github.com/postgres/postgres/commit/b7b0f3f27241e424b7103397489464d907cef2c4 Use streaming I/O in sequential scans.] -- committed in v17<br /> * [https://github.com/postgres/postgres/commit/041b96802efa33d2bc9456f2ad946976b92b5ae1 Use streaming I/O in ANALYZE.] -- committed in 17<br /> * [https://github.com/postgres/postgres/commit/3a352df05e65de740b4a375a0ecbcae97a1f6196 Use streaming I/O in pg_prewarm.] -- committed in 17<br /> * [https://commitfest.postgresql.org/48/5089/ Use streaming I/O in VACUUM] -- work in progress<br /> * [https://commitfest.postgresql.org/48/4874/ Use streaming I/O in bitmap heap scans] -- work in progress<br /> * btree [https://www.postgresql.org/message-id/flat/cf85f46f-b02f-05b2-5248-5000b894ebab%40enterprisedb.com related thread] -- todo<br /> * brin -- todo<br /> * gist -- todo<br /> * gin -- todo<br /> * CREATE DATABASE STRATEGY=WAL_LOG<br /> * recovery -- todo<br /> * ...<br /> <br /> == Users of write streams ==<br /> <br /> There are not as many places that write data out.<br /> <br /> * CHECKPOINT -- work in progress<br /> * CREATE INDEX (buffer pool bypass, extending/replacing the bulk_write.c facility) -- work in progress<br /> * VACUUM writeback -- work in progress<br /> * COPY writeback -- work in progress<br /> * eviction writeback -- work in progress<br /> <br /> = Infrastructure changes =<br /> <br /> == Read stream API ==<br /> <br /> * basic buffered read stream, single relation + fork -- committed in 17<br /> * multi-relation read stream -- todo<br /> : Needed for recovery, but also useful for CREATE DATABASE ... STRATEGY=WAL_LOG.<br /> <br /> == Write stream API ==<br /> <br /> * basic buffered write stream -- todo<br /> * driving sync_file_range -- todo<br /> * driving WAL write frequency -- todo<br /> * buffer pool bypass, needed for CREATE INDEX (bulk writes) -- todo<br /> <br /> == Committed ==<br /> <br /> * [https://commitfest.postgresql.org/33/3147/ O_DIRECT on macOS]<br /> ** So that Mac hackers can test io_data_direct=on (worker mode or posix_aio mode)<br /> * [https://commitfest.postgresql.org/31/2885/ pg_pwritev() and pg_preadv()]<br /> ** Portable support for synchronous scatter/gather I/O<br /> * [https://commitfest.postgresql.org/32/3008/ Replace buffer I/O locks with condition variables]<br /> ** So that backends other that the one that drains an I/O from the kernel can wait for it to complete<br /> * [https://commitfest.postgresql.org/41/3999/ Aligned memory allocation]<br /> ** As required for buffers used in direct I/O<br /> * [https://commitfest.postgresql.org/42/3996/ Direct I/O GUC]<br /> ** Released in 16 as debug_io_direct, to avoid attracting too much attention<br /> * [https://commitfest.postgresql.org/34/2962/ Fix DROP TABLESPACE on Windows with ProcSignalBarrier]<br /> ** This is actually an ancient bug in PostgreSQL on Windows, but it was a bit harder to hit; make check fails every time under io_method=worker because processes hang around holding cached fds.<br /> * [https://commitfest.postgresql.org/42/3993/ Refactor relation extension]<br /> ** This includes work to allow a backend to have multiple buffers in BM_IO_IN_PROGRESS state.<br /> * [https://commitfest.postgresql.org/44/4532/ Streaming I/O, vectored I/O]<br /> ** Introduce the concept of streams of buffers, initially for more efficient synchronous I/O<br /> <br /> = TODO =<br /> <br /> == General ==<br /> <br /> * partition_prune failing in CI due to compiler bug in specific MSVC release<br /> ** [https://www.postgresql.org/message-id/CAApHDvof3AEVFZTT3MArh4Tq1cAuCJb_wk8u7FV5KMzJNhU4GQ%40mail.gmail.com discussion on -hackers]<br /> ** [https://github.com/cirruslabs/docker-images-windows/issues/23 ticket to update compiler on CI] <br /> ** drop commit &quot;XXX Add temporary workaround for partition_prune test on Windows&quot;<br /> * how should flush_range op be implemented on non-Linux?<br /> * write documentation, per-OS information<br /> * can we cut down on the number of places where we do non-blocking drain, for the benefit of implementations where that might be a system call?<br /> * replication is slow with wal_sync_method=open_datasync, because we don't call WalSndWakeupRequest(); this explains why eg the subscription tests are super slow on macOS on CI (macOS defaults to open_datasync)<br /> <br /> == Larger Issues ==<br /> <br /> * Streaming read / write interface are &quot;too local&quot; to specific users. A bit more backend global awareness would likely be a good idea<br /> * currently PostgreSQL does some prefetching with posix_fadvise_WILLNEED which is not asynchronous. See [[Readahead]].<br /> <br /> == io_method=worker ==<br /> <br /> * self-adjusting IO worker pool?<br /> * more work on the spurious-wakeup vs latency tradeoff<br /> <br /> == io_method=posix_aio ==<br /> <br /> * add an &quot;interruptible&quot; field in shmem that can be used to avoid useless wakeups while the submitter is running synchronous IOs or already draining, with some double-checked flags to avoid races?<br /> * detect presence of POSIX AIO automatically so you don't have to build --with-posix-aio<br /> ** would be good to pass smoke tests on all known POSIX AIO implementations before we do that; results so far:<br /> *** Successes!<br /> **** FreeBSD<br /> **** Linux with Glibc and Musl<br /> **** illumos and Solaris<br /> **** macOS<br /> ***** need to bump up sysctl limits to get decent performance<br /> **** AIX<br /> ***** needs shared_memory_type=sysv<br /> ***** aio_nwait() would be a better wait/reap interface than aio_suspend(), but sadly it can't wait for aio_fsync()<br /> ****** possible workaround: could use aio_nwait() whenever there are no aio_fsync calls outstanding<br /> *** Failures, that will probably need to be excluded by default in early versions:<br /> **** NetBSD 9 seems to spin eating 100% CPU in lio_listio() :-(<br /> ***** clue: an earlier iteration of this code was able to pass tests on NetBSD, when we were using aio_read(), aio_write() instead of lio_listio(), and when we were using SIGEV_SIGNAL instead of SIGEV_NONE<br /> ***** no bug report filed yet, but a NetBSD developer advised me not to try to use this, it's not ready<br /> ** if keeping it as a configure option, it should be &quot;enable/disable&quot;, not &quot;with/without&quot;<br /> * fixed<br /> ** [https://cirrus-ci.com/task/6024059377942528?logs=cores#L7 crash because aio_suspend() sees EINVAL], because &quot;activate&quot; LIO IOCB too soon, [https://github.com/anarazel/postgres/commit/b174a85c446d4190cc89847760427a031711d2a1 commit (fixup to be squashed)]<br /> ** DONE: can we avoid waiting for a merge head from a later generation? yes<br /> ** DONE: can we avoid using atomics in a signal handler? most uses removed but in flight count remains; could do somethign about this<br /> ** DONE: the naming and coding in the baton stuff is weird, needs a rethink -&gt; done, now called &quot;exchange&quot; and shared with iocp<br /> ** DONE: kill the array of active IOs if not using aio_suspend (eg FreeBSD)<br /> ** DONE: would it be better to have the signal handler give up after a short time so it can get back to doing something useful, and the waiter wake it again after a bit? -&gt; seems to be better, but may need some defences against thundering herds and useless wakeups<br /> <br /> == io_method=iocp ==<br /> <br /> * Acceptance criteria for moving iocp into the main aio branch (from aio-win32):<br /> ** &quot;TRAP: FailedAssertion(&quot;pgwin32_signal_event != NULL&quot;, File: &quot;C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\src\port\open.c&quot;, Line: 78, PID: 3416)&quot;<br /> *** pgwin32_open() currently hacked to comment that out because of unresolved ordering problem, read_nondefault_variables() vs pgwin32_signal_initialize()<br /> ** currently pgaio_can_scatter_gather() considers only io_data_direct when deciding; but it applies also to WAL I/O (and potentially, in future, who knows, temporary files etc)<br /> *** this matters primarily for Windows because Windows has a different answer depending on use of direct I/O, but we have at least 3 different GUCs to control that on diffrent subsystems<br /> *** one idea would be for pgaio_can_scatter_gather() to take an IO and check the &quot;scb&quot; to see who is asking, or something like that...<br /> *** another idea is to carry a &quot;direct&quot; flag on every IO so that the merge code doesn't have to concern itself with the details beyond that<br /> ** new API: pgaio_impl-&gt;opening_fd(int fd, int flags) so that Windows impl can register fd with IOCP if flags &amp; O_OVERLAPPED? currently that's all a bit kludgy<br /> * other things<br /> ** using GetQueuedCompletionEventEx() requires Windows Vista, but PostgreSQL currently targets XP+. both are long dead, but the case for bumping it needs to be made in the community <br /> ** does FileFlushBuffers() have an async cousin? doesn't look like it<br /> *** but there is an equivalen to fdatasync() in ntdll.dll; might be worth looking further<br /> ** pgaio_iocp_closing_fd() should drain only IOs on the given fd, not all IOs issued by this backend<br /> *** compare see pgaio_posix_aio_closing_fd() -- it only drains results, it does not reap, to avoid deadlock risk! need something like that here too<br /> <br /> * fixed <br /> ** DONE: calling it &quot;iocp&quot; for now (&quot;windows&quot; was too generic; we want to reserve the option to use the new Windows io_uring knockoff API which will probably be a separate method)<br /> ** DONE: kill the IOCP thread, and teach pgaio_windows_drain() to drain?<br /> ** DONE: solve the resulting deadlock by using the same procsignal trick as posix_aio?<br /> ** DONE: we should use GetQueuedCompletionEventEx() to consume multiple completions in one call, instead of a loop!<br /> <br /> == io_method=ioring ==<br /> <br /> * no code yet, just an idea<br /> * ioring (note: no 'u', maybe should be win_ioring or some other name) is a hypothetical future IO method that would use [https://windows-internals.com/i-o-rings-when-one-i-o-operation-is-not-enough/ Windows I/O rings], a knock-off of Linux io_uring that is available in Window 11 preview but still changing<br /> * so far the documentation only describes how to do reads but [https://twitter.com/axboe/status/1435614103314128897 we know that you can already do writes and flushes with the current preview], so there is enough there right now to write aio_ioring.c and hook it up<br /> * the easiest way could be to have one ioring per backend and use aio_exchange.c to deal with cross-process problems (like io_method=posix_aio and io_method=iocp), but it may also be possible to have N iorings that are somehow shared between processes and then use the context system for interlocking (like io_method=io_uring, the Linux one); can you do that, somehow share the handle + memory mapping for the s and c queues + correct wakeups?<br /> <br /> <br /> <br /> = Quick start for PostgreSQL hackers/reviewers =<br /> <br /> * Testing the default mode, simulated AIO using io_method=worker (= the default setting)<br /> ** Try strace-ing the backend and IO worker processes to see how I/O syscalls are offloaded<br /> ** Adjust the number of io_worker processes with io_workers=N<br /> ** See the view pg_stat_aios that shows individual IOs<br /> ** See the view pg_stat_aio_backends that shows per-backend info<br /> * Testing the use of direct I/O instead of PostgreSQL's traditional double buffering<br /> ** Set io_direct=data to disable OS buffering of relation data<br /> ** Set io_direct=wal to disable OS buffering WAL data<br /> ** Set io_direct=wal,wal_init,data to disable all<br /> * Testing OS-specific options for &quot;native&quot; AIO<br /> ** Linux io_uring<br /> *** install package liburing-dev (or liburing-devel on some distros)<br /> *** configure with --with-liburing (or if using Meson, it should find it by itself?)<br /> *** run with io_method=io_uring<br /> ** POSIX AIO, on macOS, FreeBSD, NetBSD, AIX, illumos, Solaris, Linux<br /> *** configure --with-posix-aio<br /> *** run with io_method=posix_aio<br /> *** Some OS specific notes:<br /> **** Linux POSIX AIO is fake, simulated by glibc and musl with threads, and works well enough for testing but wouldn't be a good choice to actually use<br /> **** Solaris/illumos also has fake POSIX AIO (there is a more native/kernel supported AIO API but we don't support that)<br /> **** macOS has very tight limits on AIO; like every other application that uses AIO we're going to have to publish recommentations to crank them up ([http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.dc39123_1254/html/qinstmac/qinstmac11.htm Sybase for example], VirtualBox is another); on the bright side, macOS defaults provide a nice workout for the IO retry code paths; that is, often we try to start an IO and the kernel says EAGAIN, so we way for one IO to complete and then try again, and you can see this in the pg_stat_aio_backends retry counter column<br /> **** AIX only works if you set shared_memory_type=sysv because AIX's AIO can't access memory we allocate with mmap() (otherwise all IO fails with EFAULT)<br /> **** AIX direct I/O might be unnecessarily serializing per-file, which we could fix with O_CONCURRENT, O_CIO or O_CIOR<br /> **** HPUX probably also needs O_CIO to avoid serializing direct IO (native AIO not working there yet but this applies to worker mode too)<br /> ** Windows IOCP<br /> *** run with io_method=iocp<br /> *** (not yet pushed to main aio branch, find it in the aio-win32 branch until it's a little more complete...)</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Atomics&diff=41426 2025-07-15T13:52:15Z

<p>Andresfreund: </p> <hr /> <div>{| border=&quot;1&quot; cellspacing=&quot;0&quot;<br /> |-<br /> !Architecture<br /> !single-copy r/w atomicity<br /> !TAS<br /> !cmpxchg<br /> !xadd<br /> !gcc support from version<br /> !last year sold<br /> !cur. PG support<br /> !references<br /> |-<br /> !x86-32(386)<br /> |?<br /> |y (via xchgb)<br /> |n<br /> |n<br /> |4.1<br /> |2007<br /> |<br /> |not fully contained in architecture manual anymore<br /> |-<br /> !x86-32(486)<br /> |4<br /> |y (via xchgb/xadd)<br /> |4 (via cmpxchg)<br /> |4 (via xadd)<br /> |4.1<br /> |<br /> |<br /> |Intel® 64 and IA-32 Architectures Software Developer’s Manual, Vol 3A, Section 8.1.1<br /> Instruction SetReference, A-M, CMPXCHG<br /> |-<br /> !x86-32(586+)<br /> |1/2/4/8<br /> |y (via xchgb/xadd)<br /> |1/2/4/8 (via cmpxchg/cmpxchg8b)<br /> |1/2/4/8 (via xadd/cmpxchg8b)<br /> |4.1<br /> |<br /> |<br /> |Intel® 64 and IA-32 Architectures Software Developer’s Manual, Vol 3A, Section 8.1.1<br /> |-<br /> !x86-64<br /> |1/2/4/8/16<br /> |y (via xchgb/xadd)<br /> |1/2/4/8/16* (via cmpxchg/cmpxchg16b)<br /> |1/2/4/8/16* (via xadd/cmpxchg16b)<br /> |4.1<br /> |<br /> |<br /> |Intel® 64 and IA-32 Architectures Software Developer’s Manual, Vol 3A, Section 8.1.1<br /> |-<br /> !alpha<br /> |4/8<br /> |y<br /> |4/8 (via LL/SC)<br /> |4/8 (via LL/SC)<br /> |4.1<br /> |2007<br /> |gcc only<br /> |&quot;AXP architecture manual ftp://ftp.netbsd.org/pub/NetBSD/misc/dec-docs/index.html#EY-T132E-DP&quot; section 5.5.2, 5.2.2<br /> |-<br /> !arm-v5 and earlier<br /> |1/2/4?/8***<br /> |y (via SWPB)<br /> |n ***<br /> |n ***<br /> |4.6/4.4 (linux)<br /> |<br /> |gcc only<br /> |<br /> |-<br /> !arm-v6+<br /> (application profiles, w. MMU)<br /> |1/2/4/8***<br /> |y (via ldrex/strex)<br /> |4 (via ldrex/strex)<br /> |4 (via ldrex/strex)<br /> |4.6/4.4 (linux)<br /> |<br /> |gcc only<br /> |&quot;Arm Architecture Manual&quot; A3.5.3 Atomicity in the ARM architecture<br /> |-<br /> !arm-v7+<br /> (application profiles, w. MMU)<br /> |1/2/4/8***<br /> |y (via ldrex/strex)<br /> |1/2/4/8 (via ldrex/strex)<br /> |1/2/4/8 (via ldrex/strex)<br /> |4.6/4.4 (linux)<br /> |<br /> |gcc only<br /> |&quot;Arm Architecture Manual&quot; A3.5.3 Atomicity in the ARM architecture<br /> |-<br /> !powerpc<br /> |1/2/4/8<br /> |y (via lwarx/stwcx)<br /> |4/8 (via lwarx/stwcx, 8 only some versions)<br /> |4/8 (via lwarx/stwcx)<br /> |4.1<br /> |<br /> |gcc only<br /> |&quot;PowerPC Microprocessor Family: The Programming Environments for 32-Bit Microprocessors&quot;: 5.1.2 Atomicity<br /> &quot;Enhanced PowerPC™ Architecture&quot;: 6.1.3 Single-Copy Atomicity<br /> |-<br /> !64bit powerpc<br /> |1/2/4/8<br /> |y (via lwarx/stwcx,ldarx/stdcx)<br /> |1/2/4/8 (LL/SC)<br /> |1/2/4/8 (LL/SC)<br /> |4.1<br /> |<br /> |gcc only<br /> |&quot;Enhanced PowerPC™ Architecture&quot;: 6.1.3 Single-Copy Atomicity<br /> |-<br /> !s390/s390x<br /> |4/8**<br /> |y (via compare/swap)<br /> |4/8 (cs/cds)<br /> |4/8 (cs/cds)<br /> |4.1<br /> |<br /> |gcc only<br /> |&quot;z/Architecture Principles of Operation&quot;: 7-53 Compare and Swap, 5-89 Storage-Operand Consistency<br /> |-<br /> !sparcv8<br /> |1/2/4/8<br /> |y (via ldstub)<br /> |n<br /> |n<br /> |4.2<br /> |1997<br /> |gcc,sunpro<br /> |&quot;The Sparc Architecture Manual Version 8&quot; : 6.1 Basic Definitions, Overview of Model<br /> |-<br /> !sparcv9<br /> |1/2/4/8<br /> |y (via ldstub)<br /> |4/8 (via cas/casx)<br /> |4/8 (via cas/casx)<br /> |4.2<br /> |<br /> |gcc,sunpro<br /> |&quot;The Sparc Architecture Manual Version 9&quot; : D.4.1 Value Atomicity<br /> |-<br /> !mips/mipsel<br /> |4?****<br /> |y (ll/sc)<br /> |4/8 (ll/sc)<br /> |4/8 (ll/sc)<br /> |4.3<br /> |<br /> |gcc<br /> |&quot;MIPS Architecture Manual Vol I&quot; : 4.1.1.4: Loads and stores Used For Atomic Updates<br /> |-<br /> !mips64<br /> |4?/8?****<br /> |y (ll/sc)<br /> |4/8 (ll/sc, lld/scd)<br /> |4/8 (ll/sc, lld/scd)<br /> |4.3<br /> |<br /> |gcc<br /> |&quot;MIPS Architecture Manual Vol I&quot; : 4.1.1.4: Loads and stores Used For Atomic Updates<br /> |-<br /> !m68k<br /> |4?****<br /> |y<br /> |4 (via cas)<br /> |4 (via cas)<br /> |4.7<br /> |<br /> |gcc linux/netbsd???*****<br /> |&quot;m68000 Family Programmers Manual&quot;: 3.1.111 Multiprocessor instructions<br /> |-<br /> !ia64<br /> |1/2/4/8/16<br /> |y<br /> |4/8 (cmpxchg)<br /> |4/8 (xadd)<br /> |4.1<br /> | -<br /> |gcc,icc,hpux acc<br /> |&quot;Intel® Itanium® Architecture Software Developer's Manual&quot;: 4.5 Memory Datum Alignment and Atomicity; CMPXCHG8B? - Compare and Exchange<br /> |-<br /> !pa-risc<br /> |4<br /> |y (ldcwx)<br /> |n<br /> |n<br /> | -<br /> |2005/2008<br /> |gcc, hppa acc<br /> |&quot;PA-RISC 1.1. Architecture and Instruction Set Reference Manual&quot;: 4-3 Atomicity of Storage Accesses, 3-16 The Synchronization Primitive, 5.40 Load and Clear Word Indexed<br /> |-<br /> !m32r<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |requires patched glibc<br /> unmaintained website/mostly dead<br /> |http://www.linux-m32r.org/<br /> |-<br /> !VAX<br /> |<br /> |<br /> |<br /> |<br /> |1997/2005<br /> |gcc/netbsd<br /> |http://web.archive.org/web/20000815201016/http://www.compaq.com/alphaserver/vax/vax_letter_final.html<br /> |-<br /> |}<br /> &lt;nowiki&gt;*&lt;/nowiki&gt; excluding some early AMD64 models<br /> <br /> &lt;nowiki&gt;**&lt;/nowiki&gt; only a rather limited number of instructions adhere<br /> <br /> &lt;nowiki&gt;***&lt;/nowiki&gt; linux kernel emulation available<br /> <br /> &lt;nowiki&gt;****&lt;/nowiki&gt; linux and various libraries assume 4 and 8. Seems to be undocumented.<br /> <br /> &lt;nowiki&gt;*****&lt;/nowiki&gt; coldfire &lt; v4 is unsupported as PG uses tas which has been removed from coldfire CPUs (which has no MMU anyway)</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Multithreading&diff=41136 2025-05-16T15:39:38Z

<p>Andresfreund: </p> <hr /> <div>== Resources ==<br /> <br /> pgsql-hackers thread: https://www.postgresql.org/message-id/31cc6df9-53fe-3cd9-af5b-ac0d801163f4%40iki.fi<br /> <br /> PGConf.eu Presentation: https://www.postgresql.eu/events/pgconfeu2023/schedule/session/4845-multi-threaded-postgresql/<br /> <br /> Work-in-progress git branch: https://github.com/hlinnaka/postgres/tree/threading<br /> <br /> == Ongoing work ==<br /> <br /> These patches are contributing to the effort:<br /> <br /> * SendProcSignal() → SendInterrupt(), using latches: https://commitfest.postgresql.org/49/5118/<br /> <br /> <br /> Done:<br /> * replace strtok(): https://commitfest.postgresql.org/48/5071/<br /> * thread-safety: gmtime_r(), localtime_r() https://commitfest.postgresql.org/48/5084/<br /> * Refactoring postmaster's code to cleanup after child exit: https://www.postgresql.org/message-id/8f2118b9-79e3-4af7-b2c9-bd5818193ca4%40iki.fi<br /> * Remove dependency on setlocale() for collation. https://commitfest.postgresql.org/48/5023/<br /> <br /> == TODO ==<br /> <br /> === Global variables ===<br /> <br /> Global variables in PostgreSQL fall into a few different categories:<br /> <br /> * per-session state<br /> * pointers to shared memory areas<br /> * other variables that are initialized at postmaster startup and never change after that<br /> * constants<br /> <br /> Many global variables are used to hold GUCs, and they can fall into any of those categories depending on whether they're PGC_POSTMASTER or PGC_USERSET.<br /> <br /> The plan:<br /> <br /> * Add annotations to all global variables, to mark which category they fall into. Variables holding per-session state are turned into thread-local variables. The distinction between the other categories is just for documentation purposes.<br /> * Provide a tool that can list all global variables that are missing the annotations (pgguclifetimes). Add that as a compile-time check ideally. This will also be useful for extension authors to find missing annotations in extensions.<br /> ** alternative idea: could we use a simple perl script for this, like we do for the PGDLLIMPORT case?<br /> * Potential standalone value:<br /> ** &quot;modified once&quot; annotated variables could be put in a separate binary section which would improve performance<br /> ** Could help with the engineering overhead of figuring out where to initialize at postmaster startup<br /> <br /> === Extensions ===<br /> <br /> Extensions will make the transition at their own pace<br /> <br /> * Add field to control file or the PG_MODULE magic to mark whether an extension supports multi-process mode only, or multi-threaded model only, or both<br /> ** other considerations: good documentation will be key, also need to work with the extension community to build understanding and consensus<br /> <br /> * Python has traditionally been single-threaded, but see https://peps.python.org/pep-0554/<br /> ** Python &gt;= 3.12 can spawn non-shared interpreters<br /> <br /> * What about Perl, other PLs?<br /> <br /> * http://postgr.es/m/CA+TgmoYsKJnVjj94HJTSOeS1=TmAgV5L=DVTd_TYL=XUKCbfEA@mail.gmail.com<br /> <br /> * Robert volunteered to look into this at 2024.pgconf.dev<br /> <br /> === PIDs in user-facing interfaces ===<br /> <br /> A few places expose PIDs to users:<br /> <br /> * pg_terminate_backend(&lt;pid&gt;)<br /> * pg_stat_activity<br /> * query cancellation<br /> <br /> Plan:<br /> <br /> * Replace PID with a thread id.<br /> * Could use OS thread ID, but behavior isn't totally consistent across platforms<br /> * Consensus at 2024.pgconf.dev seemed to be that inventing our own 32-bit thread ID would be better<br /> * Possibly pgprocno + counter<br /> * Need to make sure we don't reuse thread IDs too quickly, else we might terminate the wrong backend.<br /> * Potential standalone value: decoupling the relationship between thread and session could enable other features<br /> <br /> * Robert volunteered to look into this at 2024.pgconf.dev<br /> <br /> === Unix Signals ===<br /> <br /> [https://wiki.postgresql.org/wiki/Signals Separate page with more details on this topic.]<br /> <br /> We use Unix signals between processes currently. It's hard to use them for inter-thread communication within the same process. When you send a signal, you send it to a process, and any thread in the process can handle it.<br /> <br /> * Refactor inter-process signals with something like Procsignal and latches<br /> <br /> * With a single-process, you cannot easily &quot;kill &lt;pid&gt;&quot; to send SIGTERM or SIGUSR1 to a single backend anymore. We can provide a &quot;pg_ctl signal&quot; command to replace that.<br /> <br /> * Description/Tasks/Components:<br /> ** timers (which use sigalarm)<br /> ** deadlock detection? (already done with waitlatch so nothing happening in signal handlers so nothing to do here?)<br /> ** Sub-sub projects<br /> *** multiplex waiting on a latch and don't rely on signals (latchification project -- replacing setlatch with interrupt driven concept)<br /> *** Abstraction for the C11 thread functions that is portable (think about C11 features and C99 compiler)<br /> <br /> * Thomas and Heikki volunteered to look into this at 2024.pgconf.dev<br /> <br /> === Replace non-thread safe library functions with thread-safe variants ===<br /> <br /> * strerror() -&gt; strerror_r()<br /> <br /> * setlocale() -&gt; uselocale()<br /> ** no uselocale on windows, workarounds are complex<br /> ** https://www.postgresql.org/message-id/CWMW5OZBWJ10.1YFLQWSUE5RE9@neon.tech<br /> ** https://www.postgresql.org/message-id/17946-3e84cb577e9551c3@postgresql.org<br /> ** http://postgr.es/m/2228884bb1f1a02614b39f71a90c94d2cc8a3a2f.camel@j-davis.com<br /> <br /> * getopt_long()<br /> ** getopt_long implementation used for windows could be used for other platforms<br /> ** rearchitect postmaster startup to change options passing<br /> <br /> * Make pg_strtok() re-entrant<br /> <br /> * not all string functions with _l have been implemented in glibc<br /> <br /> * https://www.postgresql.org/message-id/856e5ec3-879f-42ee-8258-8bcc6ec9bdea@eisentraut.org<br /> <br /> * Peter and Nathan volunteered to look into this at 2024.pgconf.dev<br /> <br /> === Misc ===<br /> <br /> * Add a GUC to enable multi-threaded mode<br /> * Replace fork() with pthread_create(). Yay!<br /> <br /> * Make virtual filedescriptors (fd.c) work with threads<br /> ** Heikki volunteered to look into this at 2024.pgconf.dev (with Andres identifying performance fallout)<br /> ** increase the limit for max number of file descriptors<br /> ** diagnose and fix the performance fallout (due to sharing VFD cache and maybe because some OSes don't cope well with large numbers of file descriptors)<br /> ** split fd.c into two files: one for virtual file descriptor abstraction and another for &quot;other&quot; file-related functions (see also src/common/file_utils.c)<br /> <br /> * Make bootparse.y and other bison/flex generated code re-entrant<br /> ** main parser is already done, but others need work, which is thought to be simple<br /> ** Peter volunteered to look into this at 2024.pgconf.dev<br /> <br /> * Refactoring guc_tables.c<br /> ** currently uses memory address of global GUC variable (and that won't work anymore with threads)<br /> <br /> * Refactor connection acceptance<br /> ** Currently the postmaster accepts connections and then spawns a process<br /> ** In the threaded model we'll want 2 processes, a supervisor that only restarts the other process, and the main process with all the threads<br /> ** But that means connection acceptance needs to be split out from supervisor tasks<br /> <br /> * verify_cb in be-secure-openssl.c uses a (static) global cert_errdetail variable to pass information. Refactor to use x509_store_ctx_get_ex_data<br /> <br /> == Open Questions ==<br /> <br /> * If an extension launches threads today, it might call SPI or other PostgreSQL functions from a different thread. That works today, if you're careful and only do that from one thread at at time. After replacing global variables with thread-local variables, that no longer works. Or did it work? We have things like longjmp() and stack depth checks that would already break that?</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_18_Open_Items&diff=40840 2025-04-01T20:23:31Z

<p>Andresfreund: </p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Use-after-free with stats_fetch_consistency = snapshot and pg_stat_get_backend_wal()<br /> ** Owner: Michael Paquier<br /> ** Commit: {{PgCommitURL|76def4cdd}}<br /> ** {{messageLink|f1788cc0-253a-4a3a-aee0-1b8ab9538736@gmail.com|Details of report}}<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * {{messageLink|flat/CAKAnmmKwiMHik5AHmBEdf5vqzbOBbcwEPHo4-PioWeAbzwcTOQ@mail.gmail.com|Enable data checksums by default}}<br /> ** Commit: {{PgCommitURL|04bec894a0}}<br /> ** Data checksums have been enabled by default. pg_upgrade cannot upgrade between non-checksum and checksum instances, so some additional steps are required by the user. Check whether the upgrade experience is tolerable and sufficiently documented. See the following list of threads that might be enhancing the end user experience:<br /> *** {{messageLink|flat/E07A611B-9CF3-4FDB-8CE8-A221E39040EC%40yesql.se|Changing the state of data checksums in a running cluster}}<br /> <br /> * {{messageLink|flat/653f3b84-fc87-45a7-9a0c-bfb4fcab3e7d%40eisentraut.org|fixing tsearch locale support}}<br /> ** Commit: {{PgCommitURL|fb1a18810f0}}<br /> ** Text search locale support has been updated and integrated into the common locale framework. Possible upgrade issues should probably at least be noted in the release notes.<br /> <br /> * Enable statistics in pg_dump by default (Jeff Davis)?<br /> ** {{messageLink|1=CAKAnmmKCgnqcd3x4Z+5yXsY7urpDS6D_NZG7pr=DYD=LHt5X4g@mail.gmail.com|2=Greg Sabino Mullane}}<br /> ** {{messageLink|1=CA+TgmoYGuEC=E4TbomRyqO8+Uav=90G0dYXw=rA1d5UmRB76Yg@mail.gmail.com|2=Robert Haas}}<br /> <br /> * decide what io_method to default to (Andres Freund)<br /> ** Commit: {{PgCommitURL|247ce06b883}}<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * {{messageLink|873c33c5-ef9e-41f6-80b2-2f5e11869f1c%40garret.ru|Incorrect results for bitmap heap scan.}}<br /> ** Certain Bitmap Heap scans only test the visibility map before returning tuples. This can conflict with VACUUM removing tuples, thus causing more tuples to be returned from the scan than what are actually visible.<br /> <br /> * {{messageLink|1=CAH2-Wz%3DPqOziyRSrnN5jAtfXWXY7-BJcHz9S355LH8Dt%3D5qxWQ@mail.gmail.com|2=Incorrect results for (SP-)GIST index-only scans}}<br /> ** Similar issue like the above.<br /> <br /> === Fixed issues ===<br /> <br /> == Non-bugs ==<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 18beta1 ===<br /> <br /> == Won't Fix ==<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Feature Freeze: April 8, 2025 0:00 AoE<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 17 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_18_Open_Items&diff=40839 2025-04-01T20:23:16Z

<p>Andresfreund: </p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Use-after-free with stats_fetch_consistency = snapshot and pg_stat_get_backend_wal()<br /> ** Owner: Michael Paquier<br /> ** Commit: {{PgCommitURL|76def4cdd}}<br /> ** {{messageLink|f1788cc0-253a-4a3a-aee0-1b8ab9538736@gmail.com|Details of report}}<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * {{messageLink|flat/CAKAnmmKwiMHik5AHmBEdf5vqzbOBbcwEPHo4-PioWeAbzwcTOQ@mail.gmail.com|Enable data checksums by default}}<br /> ** Commit: {{PgCommitURL|04bec894a0}}<br /> ** Data checksums have been enabled by default. pg_upgrade cannot upgrade between non-checksum and checksum instances, so some additional steps are required by the user. Check whether the upgrade experience is tolerable and sufficiently documented. See the following list of threads that might be enhancing the end user experience:<br /> *** {{messageLink|flat/E07A611B-9CF3-4FDB-8CE8-A221E39040EC%40yesql.se|Changing the state of data checksums in a running cluster}}<br /> <br /> * {{messageLink|flat/653f3b84-fc87-45a7-9a0c-bfb4fcab3e7d%40eisentraut.org|fixing tsearch locale support}}<br /> ** Commit: {{PgCommitURL|fb1a18810f0}}<br /> ** Text search locale support has been updated and integrated into the common locale framework. Possible upgrade issues should probably at least be noted in the release notes.<br /> <br /> * Enable statistics in pg_dump by default (Jeff Davis)?<br /> ** {{messageLink|1=CAKAnmmKCgnqcd3x4Z+5yXsY7urpDS6D_NZG7pr=DYD=LHt5X4g@mail.gmail.com|2=Greg Sabino Mullane}}<br /> ** {{messageLink|1=CA+TgmoYGuEC=E4TbomRyqO8+Uav=90G0dYXw=rA1d5UmRB76Yg@mail.gmail.com|2=Robert Haas}}<br /> <br /> * decide what io_method to default on (Andres Freund)<br /> ** Commit: {{PgCommitURL|247ce06b883}}<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * {{messageLink|873c33c5-ef9e-41f6-80b2-2f5e11869f1c%40garret.ru|Incorrect results for bitmap heap scan.}}<br /> ** Certain Bitmap Heap scans only test the visibility map before returning tuples. This can conflict with VACUUM removing tuples, thus causing more tuples to be returned from the scan than what are actually visible.<br /> <br /> * {{messageLink|1=CAH2-Wz%3DPqOziyRSrnN5jAtfXWXY7-BJcHz9S355LH8Dt%3D5qxWQ@mail.gmail.com|2=Incorrect results for (SP-)GIST index-only scans}}<br /> ** Similar issue like the above.<br /> <br /> === Fixed issues ===<br /> <br /> == Non-bugs ==<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 18beta1 ===<br /> <br /> == Won't Fix ==<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Feature Freeze: April 8, 2025 0:00 AoE<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 17 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_18_Open_Items&diff=40838 2025-04-01T20:21:32Z

<p>Andresfreund: Move open item into the right section</p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Use-after-free with stats_fetch_consistency = snapshot and pg_stat_get_backend_wal()<br /> ** Owner: Michael Paquier<br /> ** Commit: {{PgCommitURL|76def4cdd}}<br /> ** {{messageLink|f1788cc0-253a-4a3a-aee0-1b8ab9538736@gmail.com|Details of report}}<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * {{messageLink|flat/CAKAnmmKwiMHik5AHmBEdf5vqzbOBbcwEPHo4-PioWeAbzwcTOQ@mail.gmail.com|Enable data checksums by default}}<br /> ** Commit: {{PgCommitURL|04bec894a0}}<br /> ** Data checksums have been enabled by default. pg_upgrade cannot upgrade between non-checksum and checksum instances, so some additional steps are required by the user. Check whether the upgrade experience is tolerable and sufficiently documented. See the following list of threads that might be enhancing the end user experience:<br /> *** {{messageLink|flat/E07A611B-9CF3-4FDB-8CE8-A221E39040EC%40yesql.se|Changing the state of data checksums in a running cluster}}<br /> <br /> * {{messageLink|flat/653f3b84-fc87-45a7-9a0c-bfb4fcab3e7d%40eisentraut.org|fixing tsearch locale support}}<br /> ** Commit: {{PgCommitURL|fb1a18810f0}}<br /> ** Text search locale support has been updated and integrated into the common locale framework. Possible upgrade issues should probably at least be noted in the release notes.<br /> <br /> * Enable statistics in pg_dump by default (Jeff Davis)?<br /> ** {{messageLink|1=CAKAnmmKCgnqcd3x4Z+5yXsY7urpDS6D_NZG7pr=DYD=LHt5X4g@mail.gmail.com|2=Greg Sabino Mullane}}<br /> ** {{messageLink|1=CA+TgmoYGuEC=E4TbomRyqO8+Uav=90G0dYXw=rA1d5UmRB76Yg@mail.gmail.com|2=Robert Haas}}<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * {{messageLink|873c33c5-ef9e-41f6-80b2-2f5e11869f1c%40garret.ru|Incorrect results for bitmap heap scan.}}<br /> ** Certain Bitmap Heap scans only test the visibility map before returning tuples. This can conflict with VACUUM removing tuples, thus causing more tuples to be returned from the scan than what are actually visible.<br /> <br /> * {{messageLink|1=CAH2-Wz%3DPqOziyRSrnN5jAtfXWXY7-BJcHz9S355LH8Dt%3D5qxWQ@mail.gmail.com|2=Incorrect results for (SP-)GIST index-only scans}}<br /> ** Similar issue like the above.<br /> <br /> === Fixed issues ===<br /> <br /> == Non-bugs ==<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 18beta1 ===<br /> <br /> == Won't Fix ==<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Feature Freeze: April 8, 2025 0:00 AoE<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 17 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=2024.pgconf.dev_Patch_Review_Workshop&diff=39103 2024-05-30T16:10:15Z

<p>Andresfreund: /* Suggested Patches for Less Experienced Reviewers */</p> <hr /> <div>Conference: https://2024.pgconf.dev/&lt;br /&gt;<br /> Session: https://www.pgevents.ca/events/pgconfdev2024/schedule/session/40-patch-review-workshop-registration-required/<br /> <br /> Please do not add patches to the lists below. Instead, contact Robert and Andres and send them your suggestions. Thanks.<br /> <br /> == Suggested Patches for More Experienced Reviewers ==<br /> <br /> * [https://commitfest.postgresql.org/48/4476/ Extract numeric field in JSONB more effectively] by Andy Fan<br /> * [https://commitfest.postgresql.org/48/4950/ Show WAL stats on pg_stat_io] by Nazir Bilal Yavuz (reviewed by group of Bharath, Ilya, Andrey)<br /> * [https://commitfest.postgresql.org/48/4947/ Use read streams in CREATE DATABASE command when the strategy is wal_log] by Nazir Bilal Yavuz<br /> * [https://commitfest.postgresql.org/48/4518/ Support prepared statement invalidation when result or argument types change] by Jelte Fennema-Nio<br /> * [https://commitfest.postgresql.org/48/4972/ Virtual generated columns] by Peter Eisentraut<br /> * [https://commitfest.postgresql.org/48/3663/ Switching XLog source from archive to streaming when primary available] by Bharath Rupireddy<br /> * [https://commitfest.postgresql.org/48/4754/ Add LSN &lt;-&gt; time conversion facility] by Melanie Plageman (reviewed by group of Bharath, Ilya, Andrey)<br /> * [https://commitfest.postgresql.org/48/3733/ Amcheck verification of GiST and GIN] by Andrey Borodin<br /> * [https://commitfest.postgresql.org/48/4929/ apply_scanjoin_target_to_paths and partitionwise join] by Ashutosh Bapat<br /> * [https://commitfest.postgresql.org/48/4908/ Recheck foreign key if a referenced collation was nondeterministic] by Paul Jungwirth<br /> * [https://commitfest.postgresql.org/48/4884/ Fix rare recovery shutdown hang] by Thomas Munro<br /> * [https://commitfest.postgresql.org/48/3998/ SetLatches] by Thomas Munro<br /> * [https://commitfest.postgresql.org/48/4480/ Add pg_wait_for_lockers() function] by Will Mortensen<br /> <br /> == Suggested Patches for Less Experienced Reviewers ==<br /> <br /> * [https://commitfest.postgresql.org/48/4764/ Adding comments to help understand psql hidden queries] by Greg Sabino Mullane<br /> * [https://commitfest.postgresql.org/48/4951/ teach pg_combinebackup to detect deleted files where possible] by Robert Haas<br /> * [https://commitfest.postgresql.org/48/4926/ SupportRequestRows support function for generate_series_timestamptz] by David Rowley<br /> * [https://commitfest.postgresql.org/48/4932/ Add memory context type to pg_backend_memory_contexts view] by David Rowley<br /> * [https://commitfest.postgresql.org/48/4888/ Run deferred triggers in the correct security context] by Laurenz Albe<br /> * [https://commitfest.postgresql.org/48/4549/ Should consider materializing the cheapest inner path in consider_parallel_nestloop()] by Tender Wang (to review by Paul, Tomasz, Yuki)<br /> * [https://commitfest.postgresql.org/48/4553/ Reuse child_relids in try_partitionwise_join] by Ashutosh Bapat<br /> * [https://commitfest.postgresql.org/48/4964/ Reduce the log spam by excluding certain SQLSTATEs] by Laurenz Albe (Bilal, Melih, Trevor)<br /> * [https://commitfest.postgresql.org/48/4975/ pg_parse_json: don't leak token copies on failure] by Jacob Champion<br /> * [https://commitfest.postgresql.org/48/4453/ Flush disk write caches by default on macOS and Windows] by Thomas Munro<br /> * [https://commitfest.postgresql.org/48/4999/ Restrict EXPLAIN (ANALYZE) for RLS and security_barrier views] by Laurenz Albe<br /> * [https://commitfest.postgresql.org/48/4650/ Support for named parsed statement in psql] by Anthonin Bonnefoy<br /> * [https://commitfest.postgresql.org/48/3256/ functions to compute size of schemas/AMs] by Justin Pryzby (David, Stefan, Joseph)</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PgCon_2024_Developer_Meeting&diff=39070 2024-05-28T20:07:37Z

<p>Andresfreund: </p> <hr /> <div>= Minutes =<br /> <br /> Rough, I (Andres) was delegated this because I was a few minutes late :)<br /> <br /> <br /> == Operations and Governance - Jonathan Katz ==<br /> <br /> * Lots of committees aren't known even by people involved in the project<br /> * Joe Conway plans to create a committee website<br /> * Lots of independently moving parts of the project<br /> ** partially intentionally done that way, for resilience to takeover<br /> ** also an obstruction to transparency<br /> * Teams:<br /> ** Core Team - &quot;last resort&quot;, conflicts etc<br /> ** Committers - code obviously, plenty policy around that<br /> ** Pginfra - manages all the postgres infrastructure, servers for git etc<br /> ** Code of Conduct Committee - Deal with violations of the CoC<br /> ** Security 0 deal with security issues in our code<br /> *** CNA we can now assign our own CVEs<br /> ** Release - manage releases of postgres<br /> *** RMT<br /> *** Press<br /> ** Translators<br /> ** Packaging - build binary packages of postgres<br /> ** Press<br /> ** Planet<br /> ** pg.org moderation<br /> ** pgweb<br /> ** twitter etc<br /> ** Contributors team<br /> ** Funds<br /> ** GSCOC<br /> ** PGCA - Trademark etc<br /> ** Postgres Women<br /> ** Funds Groups<br /> ** Events Canada, organizing this conference<br /> ** I missed transcribing some, Jonathan missed listing some<br /> * Amazing that it works, but it also often doesn't<br /> * Robert: Really important to have list public, for volunteers to be able to oin<br /> * Karen Jax: staffing, not manning teams<br /> * Joe: Quick recap of the plans for listing of the teams on the website<br /> ** some teams already have pages, but there's no coherent way to find those<br /> ** have index with brief description and links to pages<br /> <br /> == Core Team - Tom Lane ==<br /> <br /> * At Berkeley, Postgres then Postgres 95, a few outside users<br /> * Marc Fournier volunteered to host development<br /> * Initial Core Team:<br /> ** 1996:<br /> *** Mark Fournier<br /> *** Bruce Mojian<br /> *** Bryan Henderson<br /> ** 1998:<br /> *** Jan Wiek<br /> *** Tom Lane<br /> * Initially did nearly all the work<br /> * Early additional responsibility:<br /> ** releases<br /> ** infrastructure<br /> ** committter selection<br /> * Additional responsibility<br /> ** Liaison<br /> ** Security<br /> ** Trademark<br /> ** really because ther wasn't anybody else<br /> * Core never did:<br /> ** Website<br /> ** Advocacy<br /> ** Fundraising<br /> * 2000: Landmark / Great Bridge<br /> ** started a postgres services company<br /> ** closed in .com bust<br /> ** Asked: How can this be done well with the community?<br /> *** Not enough bandwidth / experience in core *&gt; delegate<br /> ** Delegate:<br /> *** security, in 2007<br /> *** pginfra, in 2011<br /> **** took over more incrementally<br /> **** Tom doesn't remember all the details<br /> **** some small preexisting responsibilities<br /> *** Trademark issues<br /> **** PgCAC, formally incorporated NPO<br /> *** Release, in 2015<br /> **** are we ready<br /> **** do we need to release<br /> **** about 2017 we switch to time based releases, for both minor and major releases<br /> *** RMT, in 2016<br /> *** Committter selection, in 2015<br /> **** private committers list<br /> *** CoC, in 2016<br /> **** initial idea was for core to do enforcement<br /> **** now CoC does that<br /> ** One would think core doesn't have work after all that delegation<br /> ** Still lots of coordination / &quot;cat herding&quot;<br /> ** Important teams overlap with Core, CoCC is explicitly separate<br /> ** largely independent teams<br /> *** Advocacy<br /> *** Funds<br /> ** Questions:<br /> *** Stacey: Core is self selecting, plans to change that?<br /> **** Tom:<br /> ***** Not currently<br /> ***** Should definitely be long-term members for institutional memory<br /> **** Bruce: Takes a while to get there, we were looking for people a few times, didn't work out<br /> **** Robert: Self selecting is an issue, but not clear what the solution is<br /> **** Dave: It's hard to model something resilient against takeover<br /> **** Karen: At least we should be honest about this issue toward the community<br /> **** Heikki: Community is very concensus drive, hard to make decision, but also good to not piss off people. Core team has been decent at that.<br /> **** Tom: Policy that there is not more than 50% of one company<br /> **** Robert: Feels awkward to be &quot;steered&quot; by a something as opaque<br /> <br /> == Security Team - Noah ==<br /> <br /> * 17 people<br /> * receives reports of security issues<br /> * issues come from three groups<br /> ** postgres devs<br /> ** people we haven't heard of, typically something operating automated scanner<br /> *** our mailing lists are public, you know?!?<br /> ** occasionally reports by &quot;real&quot; security researchers<br /> * fixes developed by<br /> ** reports<br /> ** security team members<br /> ** &quot;outsourcing program&quot;, to non-security member committers<br /> * 7 security issues in the last year, one high risk one<br /> * We are now a CNA<br /> ** CVEs are assigned for most vulnerabilities<br /> ** new: encouragement to become our own numbering authority<br /> ** why worthwhile: CVEs assigned by outside numbering authorities by other numbering authorities<br /> ** with CNA we're supposed to be asked first<br /> ** now can update metadata for old CVEs<br /> ** also now responsible for assigning CVEs in the wider postgres space<br /> ** also get asked about filling out security questionaire<br /> * Question to audience: Problems with security model?<br /> ** Dave Cramer: Process more complicated than github's<br /> *** tarballs are released, commit messages mails are suppressed, but everyone can look at commits<br /> *** in theory we could make fix in private repo, like we did once before<br /> <br /> <br /> == PostgreSQL Community Association - Steve Singer ==<br /> <br /> * Special purpose entitity to focus on intelectual property matters<br /> * Maintain and protect assets (trademarks, domain names, not code)<br /> * does not hold copyright on postgres code, held by individuals<br /> * Canadian Not for Profit<br /> * Board:<br /> ** Dave Page<br /> ** Steve Siunger<br /> ** Jonathan Katz<br /> ** Jaime Casanova<br /> ** Peter Eisentraut<br /> ** Marc Fournier<br /> * self selecting membership, by board (past and present members)<br /> ** aware of issues around that<br /> * Why:<br /> ** ensure fair use of PostgreSQL brand assets<br /> ** otherwise somebody else can call something postgres<br /> ** requires enforcement, otherwise trademark is abandoned and can't be enforced<br /> ** keep PG fre and open source<br /> * Where:<br /> ** Canada<br /> ** US<br /> ** Europe<br /> * How:<br /> ** controlled fair use allowed<br /> ** example: Brand names involving postgres need to be licensed<br /> ** standard license for templates, but often companies want modified terms<br /> * Key Activities:<br /> ** Oppose Trademark Registrations<br /> *** some conflicts around that are public, others not<br /> ** Ask that the trademarks be used properly<br /> ** License where needed<br /> * Highlights<br /> ** Fundacion PostgreSQL conclusion (after three years and lots of money)<br /> ** Trademark oppositions, cease &amp; desists<br /> ** Donor Recognition Program<br /> * Why private?<br /> ** public statement can have legal consequences, need to be reviewed by lawyers, costs money for review<br /> ** rather work amiciably, rather than public shaming, often succeeds<br /> ** some unlicensed uses are because we didn't have the time/money to do something about them<br /> * Financial highlights:<br /> ** Net loss: CAD 11k<br /> ** Goal: Improve financials via Donor recognition program<br /> ** Various levels that grant public recognition<br /> * Question:<br /> <br /> <br /> == Code of Conduct Committee - Eliza Benett ==<br /> <br /> * Eliza is chair since 2014<br /> * Overview of the last year<br /> ** Representation:<br /> *** 5 members, 3 male identifying, 2 female, plus chair<br /> *** geographic spread: 2 from Canada, 1 from Asial, one american in Asia, 1 from Europe<br /> *** Volunteers, selected by current members<br /> *** How can group size be increased<br /> ** Activity<br /> *** 8 complaints in 2023/24 (inappropriate conduct on community channels (2), at community events (2), non-community channel (3))<br /> *** Outcomes: no violation (3), out of scope (2), corrective recommended and taken (2), corrective action recommended (1)<br /> *** processing done by lead investigator, always with at least one other person<br /> *** [couldn't keep up]<br /> ** Trends:<br /> *** 2023/24 have been busy<br /> *** Complaints on social media, [couldn't keep up]<br /> ** Opportunities:<br /> *** need new members<br /> *** update CoC policy, hasn't bene updated significantly since introduction<br /> *** improve enforcement, appeals process<br /> *** inform community events about CoC existance / use of CoC[C]<br /> ** Revisions:<br /> *** Roles &amp; responsibilities<br /> *** Scope clarification<br /> *** Appeals process<br /> *** Record sharing and retention, records exist since 2019 (unsure)<br /> *** Communication processes<br /> *** ....<br /> *** proposed timeline: Summer 2024, engegement on policy revisions, September: 2024, in effect<br /> <br /> <br /> == Port report - Thomas Munro ==<br /> <br /> * lots of platform that postgres used to run on, but doesn't anymore<br /> * still support about 11 platforms<br /> * most users on linux, some on windows, the rest probably miniscule<br /> * Robert Haas and Devrim Gunduz think there are way more on windows than Thomas guessed<br /> * Dave Page says some windows apps embed postgres<br /> * Estimate about problem per platform, overwhelmingly windows<br /> ** Robert: Lots of it just perl crap<br /> * Highlights:<br /> ** AIX was dropped in 2017, suddenly IBM folks appeared, rely on Postgres<br /> *** AIX is weird, unusual linker/object format<br /> *** Old OS versions in buildfarm<br /> *** Vendor compiler requires extra work<br /> **** some bugs got fixed<br /> ** HP-UX was dropped in 16<br /> *** no real complaints, one mention<br /> ** Solaris &lt;= 11.3 in 17 was dropped<br /> *** solaris is less painful<br /> *** user interest, solaris buildfarm animals by users<br /> *** also illumos<br /> *** they stopped increasing versions after 11.4<br /> **** should we drop support for the sunpro compiler?<br /> *** won't support newer things like direct-io<br /> ** FreeBSD:<br /> *** Short supported version window, buildfarm animals exist that are very old<br /> ** OpenBSD:<br /> *** uses libressl<br /> ** Linux+musl<br /> *** different libc, a few incompatibilities that we have to deal with<br /> *** Have a buildfarm animal now<br /> ** Windows:<br /> *** Removed old perl based build system, did make things easier<br /> *** Thomas never used windows, despite working for Microsoft<br /> *** Locale support is very instable, breaks frequently<br /> **** particularly with turkish locale, breaks all PG databases with the locale, but also some others<br /> *** truncations fail occasionally<br /> **** causes corruptions<br /> *** Socket event loss, causes weird problems<br /> *** Thomas tries to fix things, but doesn't use windows, hard to find testing<br /> *** Thomas finds our state of windows support embarassing<br /> *** We ought to make some decisions to improve the situation, requires making dropping some filesystems<br /> **** who makes decision?<br /> <br /> <br /> == Group 1 ==<br /> <br /> * Action item: bugtracker<br /> ** Presented by Matthias Van de Meent<br /> ** would be useful to have a bug-manager responsibility, likely longer than a CFM<br /> ** no agreement: add feature to track a commitfest entry for threads without patches<br /> ** Same topic was also discussed by Group 4 (Peter Eisentraut)<br /> *** We agreed that we should have one<br /> *** Who would curate? Hire somebody?<br /> *** What software to use? Gitlab?<br /> ** Melanie: Is it it really a technical problem?<br /> ** Andres: We seem to have started having agreement on having a bugtracker, that's different than in the past<br /> ** Heikki: Some tracking would be useful, even if dumping ground<br /> ** Robert: Relying on bugtracking / CF tracking person doing all the work on summarizing huge threads can't scale<br /> ** Melanie: Fixing bugs is more important than new features<br /> ** Bharat: Authors should summarize state in commit messages, several agreeing with that (Noah, Steve, Robert)<br /> ** Robert: How to balance responsiblity to deal with bugreports and feature work<br /> ** Melanie: Responsiblity of somebody doing triage would not be in-depth work, just high level triage<br /> ** Alvaro: Just wishing there weren't bug doesn't work<br /> ** Jelte: Knowing about many people are affected is useful<br /> <br /> * Action item: tooling<br /> ** More developer oriented tooling are required, including things like pull request<br /> ** Platforms like github are problematic due to restrictions imposed by companies (e.g. github was restricted in Iran for a while)<br /> ** we could use a hosted gitlab, or hosted by other open source project<br /> ** Matthias: Platform notifications are hard to use<br /> ** Robert: Real problem is changing the workflow<br /> ** Andreas: We could have patch review events, to get more people involved?<br /> ** Jonathan: Similar to python sprints<br /> ** Robert: That's likely not solving problems with complicated patches<br /> ** Andreas: Could do something like docs on a separate platform<br /> <br /> <br /> == Group 4 ==<br /> <br /> * Presented by Peter Eisentraut<br /> <br /> * Action item: Windows port<br /> ** Few / no core developers really care about windows port<br /> ** Users are less sophisticated and communicate less<br /> ** Andrew Dunstan would look at one of the pending technical issues<br /> ** Problem areas<br /> *** locale support - default to windows?<br /> *** what's with refs - it doesn't support posix mode<br /> ** Heikki: Should we just drop windows, similar to AIX<br /> ** Robert: Bugs like the windows corruption issue is not enough reason to drop support, sometimes we just have bugs<br /> <br /> * Action item: Freeze policy<br /> ** Peter and Matthias<br /> ** Group thought that this year was an anomaly, doesn't necessarily need a large scale change<br /> ** Encourage more incremental work, don't require perfection, leading to pushing very late in cycle<br /> *** Prioritize architecture of details like default values<br /> ** Impact on feature freeze on release cycle<br /> *** Branch earlier, to not have to postpone as much, group didn't really have agreement<br /> *** Melanie: Issues with backporting, focus on stability<br /> *** Matthias: Doesn't see a point reviewing during feature freeze<br /> *** Robert: Committer authored patches are part of the problem, authors have control over time<br /> *** Vik Faering: Once stable branch is forked, nobody cares about stable branch<br /> *** Tom: Agrees<br /> *** Matthias: Can strike balance, by moving branch up a month or such<br /> *** Robert: One question is if there are more reverts, that's bad after the branch<br /> *** Jonathan: Not convinced that that's really the issue, we had important fixes late<br /> *** Robert: Not really addressing the issue of large stuff late in the cycle<br /> *** Matthias: Still a problem not being able to get anything in<br /> *** Melanie: Don't think many agree with you<br /> <br /> * Action item: yubikeys for committing<br /> ** no agreement on forcing, but encouraging is ok<br /> ** laptop security etc would be good<br /> <br /> * Action item: Tooling<br /> ** Should be easier to find patches that e.g. have tests that never worked, similar tooling things<br /> <br /> <br /> == Group 2 ==<br /> <br /> * Presented by: Andreas Scherbaum<br /> <br /> * Action item: Exhibition Effort<br /> ** we're not showing up at enough non-PG conferences<br /> ** Jonathan: PGUS has some funding, but are not covering that much<br /> ** More efforts to support groups like pgWomen<br /> ** consider having an application developer oriented conferences<br /> <br /> *** Stacey: At conferences like djangocon people say they like postgres, but feel like postgres conferes aren't &quot;for them&quot;.<br /> *** Stacey: Perhaps collaborating with conferences like djangocon would help reach people and then get them to postgres<br /> ** People should submit talks at more conferes<br /> <br /> * Action item: SPI<br /> <br /> ** Jonathan: SPI is opaque, we don't know how much money we have, we get money from donations<br /> ** Heikki: Why is SPI not funding pgca<br /> ** Steve: SPI is very complicated to deal with, partially due to being out of the country, partially just SPI being SPI<br /> ** Jonathan: SPI didn't give money for pgconf.dev, that should have been easy<br /> ** Alvaro: Could SPI pay for a bug manager, agreement in audience<br /> ** Robert: Companies could also provide staff tracking for <br /> <br /> * Action item: Benchmarking<br /> ** TPC is still relevant<br /> ** Better testing needed<br /> ** Find way to run meaningful test with every relevant patch<br /> ** Jonathan: There's lots of money in the postgres world, there should be money for better testing<br /> <br /> * Action item: Commitfest<br /> ** Not enough reviewers, need more<br /> ** Idea: review events<br /> <br /> <br /> == Group 3 ==<br /> <br /> * Presented by: Jeff Davis<br /> <br /> * Action item: Protocol changes<br /> ** Which features do we want?<br /> ** How do we introduce changes, dealing with breakages?<br /> *** new protocol version?<br /> *** optional features?<br /> ** Desired features<br /> *** Compression<br /> *** Proxies and conneciton poolers<br /> *** Transparent column encryption<br /> *** Pgpool would like to have sequence numbers<br /> ** Required vs optional features<br /> *** Depends on implementation complexity on the driver side?<br /> ** Concerns around<br /> *** too partial solutions<br /> *** genericity introducing unnecessary roundtrips<br /> *** concerns about breakages<br /> ** Robert: How do we put various efforts into something coherent<br /> <br /> * Action item: Surprising behavior around roles and privileges<br /> ** Some of this is user hostile<br /> ** Users can't control what code is executed with their privileges<br /> ** Function authors don't know what their function will do<br /> *** requires setting search_path, but most won't<br /> ** There's no way to drop privileges<br /> *** SET SESSION AUTHORIZATION and SET ROLE can be undone via RESET SESSION AUTHORIZATON<br /> ** Hacks and special cases<br /> *** e.g. around table maintenance tasks like REINDEX and logical replication<br /> **** it's safe to reindex, but not safe to insert<br /> ** We define a lot of these to not be security issues, but they often are for users.<br /> *** We don't even document much of this<br /> ** Noah: Is it really a problem, we document this?<br /> ** Jeff: Yes, we document this just for security definer, but it's also a problem in other situations<br /> ** Noah: Now agree<br /> ** Concensus: People other than Robert and Jeff need to get involved, including at some point agreeing to do something</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PgCon_2024_Developer_Meeting&diff=39069 2024-05-28T20:07:00Z

<p>Andresfreund: </p> <hr /> <div>= Minutes =<br /> <br /> Rough, I (Andres) was delegated this because I was a few minutes late :)<br /> <br /> <br /> == Operations and Governance - Jonathan Katz ==<br /> <br /> * Lots of committees aren't known even by people involved in the project<br /> * Joe Conway plans to create a committee website<br /> * Lots of independently moving parts of the project<br /> ** partially intentionally done that way, for resilience to takeover<br /> ** also an obstruction to transparency<br /> * Teams:<br /> ** Core Team - &quot;last resort&quot;, conflicts etc<br /> ** Committers - code obviously, plenty policy around that<br /> ** Pginfra - manages all the postgres infrastructure, servers for git etc<br /> ** Code of Conduct Committee * Deal with violations of the CoC<br /> ** Security 0 deal with security issues in our code<br /> *** CNA we can now assign our own CVEs<br /> ** Release - manage releases of postgres<br /> *** RMT<br /> *** Press<br /> ** Translators<br /> ** Packaging - build binary packages of postgres<br /> ** Press<br /> ** Planet<br /> ** pg.org moderation<br /> ** pgweb<br /> ** twitter etc<br /> ** Contributors team<br /> ** Funds<br /> ** GSCOC<br /> ** PGCA * Trademark etc<br /> ** Postgres Women<br /> ** Funds Groups<br /> ** Events Canada, organizing this conference<br /> ** I missed transcribing some, Jonathan missed listing some<br /> * Amazing that it works, but it also often doesn't<br /> * Robert: Really important to have list public, for volunteers to be able to oin<br /> * Karen Jax: staffing, not manning teams<br /> * Joe: Quick recap of the plans for listing of the teams on the website<br /> ** some teams already have pages, but there's no coherent way to find those<br /> ** have index with brief description and links to pages<br /> <br /> == Core Team - Tom Lane ==<br /> <br /> * At Berkeley, Postgres then Postgres 95, a few outside users<br /> * Marc Fournier volunteered to host development<br /> * Initial Core Team:<br /> ** 1996:<br /> *** Mark Fournier<br /> *** Bruce Mojian<br /> *** Bryan Henderson<br /> ** 1998:<br /> *** Jan Wiek<br /> *** Tom Lane<br /> * Initially did nearly all the work<br /> * Early additional responsibility:<br /> ** releases<br /> ** infrastructure<br /> ** committter selection<br /> * Additional responsibility<br /> ** Liaison<br /> ** Security<br /> ** Trademark<br /> ** really because ther wasn't anybody else<br /> * Core never did:<br /> ** Website<br /> ** Advocacy<br /> ** Fundraising<br /> * 2000: Landmark / Great Bridge<br /> ** started a postgres services company<br /> ** closed in .com bust<br /> ** Asked: How can this be done well with the community?<br /> *** Not enough bandwidth / experience in core *&gt; delegate<br /> ** Delegate:<br /> *** security, in 2007<br /> *** pginfra, in 2011<br /> **** took over more incrementally<br /> **** Tom doesn't remember all the details<br /> **** some small preexisting responsibilities<br /> *** Trademark issues<br /> **** PgCAC, formally incorporated NPO<br /> *** Release, in 2015<br /> **** are we ready<br /> **** do we need to release<br /> **** about 2017 we switch to time based releases, for both minor and major releases<br /> *** RMT, in 2016<br /> *** Committter selection, in 2015<br /> **** private committers list<br /> *** CoC, in 2016<br /> **** initial idea was for core to do enforcement<br /> **** now CoC does that<br /> ** One would think core doesn't have work after all that delegation<br /> ** Still lots of coordination / &quot;cat herding&quot;<br /> ** Important teams overlap with Core, CoCC is explicitly separate<br /> ** largely independent teams<br /> *** Advocacy<br /> *** Funds<br /> ** Questions:<br /> *** Stacey: Core is self selecting, plans to change that?<br /> **** Tom:<br /> ***** Not currently<br /> ***** Should definitely be long-term members for institutional memory<br /> **** Bruce: Takes a while to get there, we were looking for people a few times, didn't work out<br /> **** Robert: Self selecting is an issue, but not clear what the solution is<br /> **** Dave: It's hard to model something resilient against takeover<br /> **** Karen: At least we should be honest about this issue toward the community<br /> **** Heikki: Community is very concensus drive, hard to make decision, but also good to not piss off people. Core team has been decent at that.<br /> **** Tom: Policy that there is not more than 50% of one company<br /> **** Robert: Feels awkward to be &quot;steered&quot; by a something as opaque<br /> <br /> == Security Team - Noah ==<br /> <br /> * 17 people<br /> * receives reports of security issues<br /> * issues come from three groups<br /> ** postgres devs<br /> ** people we haven't heard of, typically something operating automated scanner<br /> *** our mailing lists are public, you know?!?<br /> ** occasionally reports by &quot;real&quot; security researchers<br /> * fixes developed by<br /> ** reports<br /> ** security team members<br /> ** &quot;outsourcing program&quot;, to non-security member committers<br /> * 7 security issues in the last year, one high risk one<br /> * We are now a CNA<br /> ** CVEs are assigned for most vulnerabilities<br /> ** new: encouragement to become our own numbering authority<br /> ** why worthwhile: CVEs assigned by outside numbering authorities by other numbering authorities<br /> ** with CNA we're supposed to be asked first<br /> ** now can update metadata for old CVEs<br /> ** also now responsible for assigning CVEs in the wider postgres space<br /> ** also get asked about filling out security questionaire<br /> * Question to audience: Problems with security model?<br /> ** Dave Cramer: Process more complicated than github's<br /> *** tarballs are released, commit messages mails are suppressed, but everyone can look at commits<br /> *** in theory we could make fix in private repo, like we did once before<br /> <br /> <br /> == PostgreSQL Community Association - Steve Singer ==<br /> <br /> * Special purpose entitity to focus on intelectual property matters<br /> * Maintain and protect assets (trademarks, domain names, not code)<br /> * does not hold copyright on postgres code, held by individuals<br /> * Canadian Not for Profit<br /> * Board:<br /> ** Dave Page<br /> ** Steve Siunger<br /> ** Jonathan Katz<br /> ** Jaime Casanova<br /> ** Peter Eisentraut<br /> ** Marc Fournier<br /> * self selecting membership, by board (past and present members)<br /> ** aware of issues around that<br /> * Why:<br /> ** ensure fair use of PostgreSQL brand assets<br /> ** otherwise somebody else can call something postgres<br /> ** requires enforcement, otherwise trademark is abandoned and can't be enforced<br /> ** keep PG fre and open source<br /> * Where:<br /> ** Canada<br /> ** US<br /> ** Europe<br /> * How:<br /> ** controlled fair use allowed<br /> ** example: Brand names involving postgres need to be licensed<br /> ** standard license for templates, but often companies want modified terms<br /> * Key Activities:<br /> ** Oppose Trademark Registrations<br /> *** some conflicts around that are public, others not<br /> ** Ask that the trademarks be used properly<br /> ** License where needed<br /> * Highlights<br /> ** Fundacion PostgreSQL conclusion (after three years and lots of money)<br /> ** Trademark oppositions, cease &amp; desists<br /> ** Donor Recognition Program<br /> * Why private?<br /> ** public statement can have legal consequences, need to be reviewed by lawyers, costs money for review<br /> ** rather work amiciably, rather than public shaming, often succeeds<br /> ** some unlicensed uses are because we didn't have the time/money to do something about them<br /> * Financial highlights:<br /> ** Net loss: CAD 11k<br /> ** Goal: Improve financials via Donor recognition program<br /> ** Various levels that grant public recognition<br /> * Question:<br /> <br /> <br /> == Code of Conduct Committee - Eliza Benett ==<br /> <br /> * Eliza is chair since 2014<br /> * Overview of the last year<br /> ** Representation:<br /> *** 5 members, 3 male identifying, 2 female, plus chair<br /> *** geographic spread: 2 from Canada, 1 from Asial, one american in Asia, 1 from Europe<br /> *** Volunteers, selected by current members<br /> *** How can group size be increased<br /> ** Activity<br /> *** 8 complaints in 2023/24 (inappropriate conduct on community channels (2), at community events (2), non-community channel (3))<br /> *** Outcomes: no violation (3), out of scope (2), corrective recommended and taken (2), corrective action recommended (1)<br /> *** processing done by lead investigator, always with at least one other person<br /> *** [couldn't keep up]<br /> ** Trends:<br /> *** 2023/24 have been busy<br /> *** Complaints on social media, [couldn't keep up]<br /> ** Opportunities:<br /> *** need new members<br /> *** update CoC policy, hasn't bene updated significantly since introduction<br /> *** improve enforcement, appeals process<br /> *** inform community events about CoC existance / use of CoC[C]<br /> ** Revisions:<br /> *** Roles &amp; responsibilities<br /> *** Scope clarification<br /> *** Appeals process<br /> *** Record sharing and retention, records exist since 2019 (unsure)<br /> *** Communication processes<br /> *** ....<br /> *** proposed timeline: Summer 2024, engegement on policy revisions, September: 2024, in effect<br /> <br /> <br /> == Port report - Thomas Munro ==<br /> <br /> * lots of platform that postgres used to run on, but doesn't anymore<br /> * still support about 11 platforms<br /> * most users on linux, some on windows, the rest probably miniscule<br /> * Robert Haas and Devrim Gunduz think there are way more on windows than Thomas guessed<br /> * Dave Page says some windows apps embed postgres<br /> * Estimate about problem per platform, overwhelmingly windows<br /> ** Robert: Lots of it just perl crap<br /> * Highlights:<br /> ** AIX was dropped in 2017, suddenly IBM folks appeared, rely on Postgres<br /> *** AIX is weird, unusual linker/object format<br /> *** Old OS versions in buildfarm<br /> *** Vendor compiler requires extra work<br /> **** some bugs got fixed<br /> ** HP-UX was dropped in 16<br /> *** no real complaints, one mention<br /> ** Solaris &lt;= 11.3 in 17 was dropped<br /> *** solaris is less painful<br /> *** user interest, solaris buildfarm animals by users<br /> *** also illumos<br /> *** they stopped increasing versions after 11.4<br /> **** should we drop support for the sunpro compiler?<br /> *** won't support newer things like direct-io<br /> ** FreeBSD:<br /> *** Short supported version window, buildfarm animals exist that are very old<br /> ** OpenBSD:<br /> *** uses libressl<br /> ** Linux+musl<br /> *** different libc, a few incompatibilities that we have to deal with<br /> *** Have a buildfarm animal now<br /> ** Windows:<br /> *** Removed old perl based build system, did make things easier<br /> *** Thomas never used windows, despite working for Microsoft<br /> *** Locale support is very instable, breaks frequently<br /> **** particularly with turkish locale, breaks all PG databases with the locale, but also some others<br /> *** truncations fail occasionally<br /> **** causes corruptions<br /> *** Socket event loss, causes weird problems<br /> *** Thomas tries to fix things, but doesn't use windows, hard to find testing<br /> *** Thomas finds our state of windows support embarassing<br /> *** We ought to make some decisions to improve the situation, requires making dropping some filesystems<br /> **** who makes decision?<br /> <br /> <br /> == Group 1 ==<br /> <br /> * Action item: bugtracker<br /> ** Presented by Matthias Van de Meent<br /> ** would be useful to have a bug-manager responsibility, likely longer than a CFM<br /> ** no agreement: add feature to track a commitfest entry for threads without patches<br /> ** Same topic was also discussed by Group 4 (Peter Eisentraut)<br /> *** We agreed that we should have one<br /> *** Who would curate? Hire somebody?<br /> *** What software to use? Gitlab?<br /> ** Melanie: Is it it really a technical problem?<br /> ** Andres: We seem to have started having agreement on having a bugtracker, that's different than in the past<br /> ** Heikki: Some tracking would be useful, even if dumping ground<br /> ** Robert: Relying on bugtracking / CF tracking person doing all the work on summarizing huge threads can't scale<br /> ** Melanie: Fixing bugs is more important than new features<br /> ** Bharat: Authors should summarize state in commit messages, several agreeing with that (Noah, Steve, Robert)<br /> ** Robert: How to balance responsiblity to deal with bugreports and feature work<br /> ** Melanie: Responsiblity of somebody doing triage would not be in-depth work, just high level triage<br /> ** Alvaro: Just wishing there weren't bug doesn't work<br /> ** Jelte: Knowing about many people are affected is useful<br /> <br /> * Action item: tooling<br /> ** More developer oriented tooling are required, including things like pull request<br /> ** Platforms like github are problematic due to restrictions imposed by companies (e.g. github was restricted in Iran for a while)<br /> ** we could use a hosted gitlab, or hosted by other open source project<br /> ** Matthias: Platform notifications are hard to use<br /> ** Robert: Real problem is changing the workflow<br /> ** Andreas: We could have patch review events, to get more people involved?<br /> ** Jonathan: Similar to python sprints<br /> ** Robert: That's likely not solving problems with complicated patches<br /> ** Andreas: Could do something like docs on a separate platform<br /> <br /> <br /> == Group 4 ==<br /> <br /> * Presented by Peter Eisentraut<br /> <br /> * Action item: Windows port<br /> ** Few / no core developers really care about windows port<br /> ** Users are less sophisticated and communicate less<br /> ** Andrew Dunstan would look at one of the pending technical issues<br /> ** Problem areas<br /> *** locale support - default to windows?<br /> *** what's with refs - it doesn't support posix mode<br /> ** Heikki: Should we just drop windows, similar to AIX<br /> ** Robert: Bugs like the windows corruption issue is not enough reason to drop support, sometimes we just have bugs<br /> <br /> * Action item: Freeze policy<br /> ** Peter and Matthias<br /> ** Group thought that this year was an anomaly, doesn't necessarily need a large scale change<br /> ** Encourage more incremental work, don't require perfection, leading to pushing very late in cycle<br /> *** Prioritize architecture of details like default values<br /> ** Impact on feature freeze on release cycle<br /> *** Branch earlier, to not have to postpone as much, group didn't really have agreement<br /> *** Melanie: Issues with backporting, focus on stability<br /> *** Matthias: Doesn't see a point reviewing during feature freeze<br /> *** Robert: Committer authored patches are part of the problem, authors have control over time<br /> *** Vik Faering: Once stable branch is forked, nobody cares about stable branch<br /> *** Tom: Agrees<br /> *** Matthias: Can strike balance, by moving branch up a month or such<br /> *** Robert: One question is if there are more reverts, that's bad after the branch<br /> *** Jonathan: Not convinced that that's really the issue, we had important fixes late<br /> *** Robert: Not really addressing the issue of large stuff late in the cycle<br /> *** Matthias: Still a problem not being able to get anything in<br /> *** Melanie: Don't think many agree with you<br /> <br /> * Action item: yubikeys for committing<br /> ** no agreement on forcing, but encouraging is ok<br /> ** laptop security etc would be good<br /> <br /> * Action item: Tooling<br /> ** Should be easier to find patches that e.g. have tests that never worked, similar tooling things<br /> <br /> <br /> == Group 2 ==<br /> <br /> * Presented by: Andreas Scherbaum<br /> <br /> * Action item: Exhibition Effort<br /> ** we're not showing up at enough non-PG conferences<br /> ** Jonathan: PGUS has some funding, but are not covering that much<br /> ** More efforts to support groups like pgWomen<br /> ** consider having an application developer oriented conferences<br /> <br /> *** Stacey: At conferences like djangocon people say they like postgres, but feel like postgres conferes aren't &quot;for them&quot;.<br /> *** Stacey: Perhaps collaborating with conferences like djangocon would help reach people and then get them to postgres<br /> ** People should submit talks at more conferes<br /> <br /> * Action item: SPI<br /> <br /> ** Jonathan: SPI is opaque, we don't know how much money we have, we get money from donations<br /> ** Heikki: Why is SPI not funding pgca<br /> ** Steve: SPI is very complicated to deal with, partially due to being out of the country, partially just SPI being SPI<br /> ** Jonathan: SPI didn't give money for pgconf.dev, that should have been easy<br /> ** Alvaro: Could SPI pay for a bug manager, agreement in audience<br /> ** Robert: Companies could also provide staff tracking for <br /> <br /> * Action item: Benchmarking<br /> ** TPC is still relevant<br /> ** Better testing needed<br /> ** Find way to run meaningful test with every relevant patch<br /> ** Jonathan: There's lots of money in the postgres world, there should be money for better testing<br /> <br /> * Action item: Commitfest<br /> ** Not enough reviewers, need more<br /> ** Idea: review events<br /> <br /> <br /> == Group 3 ==<br /> <br /> * Presented by: Jeff Davis<br /> <br /> * Action item: Protocol changes<br /> ** Which features do we want?<br /> ** How do we introduce changes, dealing with breakages?<br /> *** new protocol version?<br /> *** optional features?<br /> ** Desired features<br /> *** Compression<br /> *** Proxies and conneciton poolers<br /> *** Transparent column encryption<br /> *** Pgpool would like to have sequence numbers<br /> ** Required vs optional features<br /> *** Depends on implementation complexity on the driver side?<br /> ** Concerns around<br /> *** too partial solutions<br /> *** genericity introducing unnecessary roundtrips<br /> *** concerns about breakages<br /> ** Robert: How do we put various efforts into something coherent<br /> <br /> * Action item: Surprising behavior around roles and privileges<br /> ** Some of this is user hostile<br /> ** Users can't control what code is executed with their privileges<br /> ** Function authors don't know what their function will do<br /> *** requires setting search_path, but most won't<br /> ** There's no way to drop privileges<br /> *** SET SESSION AUTHORIZATION and SET ROLE can be undone via RESET SESSION AUTHORIZATON<br /> ** Hacks and special cases<br /> *** e.g. around table maintenance tasks like REINDEX and logical replication<br /> **** it's safe to reindex, but not safe to insert<br /> ** We define a lot of these to not be security issues, but they often are for users.<br /> *** We don't even document much of this<br /> ** Noah: Is it really a problem, we document this?<br /> ** Jeff: Yes, we document this just for security definer, but it's also a problem in other situations<br /> ** Noah: Now agree<br /> ** Concensus: People other than Robert and Jeff need to get involved, including at some point agreeing to do something</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PgCon_2024_Developer_Meeting&diff=39068 2024-05-28T17:39:19Z

<p>Andresfreund: Created page with &quot;= Minutes = Rough, I (Andres) was delegated this because I was a few minutes late :) == Operations and Governance - Jonathan Katz == * Lots of committees aren&#039;t known even...&quot;</p> <hr /> <div>= Minutes =<br /> <br /> Rough, I (Andres) was delegated this because I was a few minutes late :)<br /> <br /> <br /> == Operations and Governance - Jonathan Katz ==<br /> <br /> * Lots of committees aren't known even by people involved in the project<br /> * Joe Conway plans to create a committee website<br /> * Lots of independently moving parts of the project<br /> ** partially intentionally done that way, for resilience to takeover<br /> ** also an obstruction to transparency<br /> * Teams:<br /> ** Core Team - &quot;last resort&quot;, conflicts etc<br /> ** Committers - code obviously, plenty policy around that<br /> ** Pginfra - manages all the postgres infrastructure, servers for git etc<br /> ** Code of Conduct Committee * Deal with violations of the CoC<br /> ** Security 0 deal with security issues in our code<br /> *** CNA we can now assign our own CVEs<br /> ** Release - manage releases of postgres<br /> *** RMT<br /> *** Press<br /> ** Translators<br /> ** Packaging - build binary packages of postgres<br /> ** Press<br /> ** Planet<br /> ** pg.org moderation<br /> ** pgweb<br /> ** twitter etc<br /> ** Contributors team<br /> ** Funds<br /> ** GSCOC<br /> ** PGCA * Trademark etc<br /> ** Postgres Women<br /> ** Funds Groups<br /> ** Events Canada, organizing this conference<br /> ** I missed transcribing some, Jonathan missed listing some<br /> * Amazing that it works, but it also often doesn't<br /> * Robert: Really important to have list public, for volunteers to be able to oin<br /> * Karen Jax: staffing, not manning teams<br /> * Joe: Quick recap of the plans for listing of the teams on the website<br /> ** some teams already have pages, but there's no coherent way to find those<br /> ** have index with brief description and links to pages<br /> <br /> == Core Team - Tom Lane ==<br /> <br /> * At Berkeley, Postgres then Postgres 95, a few outside users<br /> * Marc Fournier volunteered to host development<br /> * Initial Core Team:<br /> ** 1996:<br /> *** Mark Fournier<br /> *** Bruce Mojian<br /> *** Bryan Henderson<br /> ** 1998:<br /> *** Jan Wiek<br /> *** Tom Lane<br /> * Initially did nearly all the work<br /> * Early additional responsibility:<br /> ** releases<br /> ** infrastructure<br /> ** committter selection<br /> * Additional responsibility<br /> ** Liaison<br /> ** Security<br /> ** Trademark<br /> ** really because ther wasn't anybody else<br /> * Core never did:<br /> ** Website<br /> ** Advocacy<br /> ** Fundraising<br /> * 2000: Landmark / Great Bridge<br /> ** started a postgres services company<br /> ** closed in .com bust<br /> ** Asked: How can this be done well with the community?<br /> *** Not enough bandwidth / experience in core *&gt; delegate<br /> ** Delegate:<br /> *** security, in 2007<br /> *** pginfra, in 2011<br /> **** took over more incrementally<br /> **** Tom doesn't remember all the details<br /> **** some small preexisting responsibilities<br /> *** Trademark issues<br /> **** PgCAC, formally incorporated NPO<br /> *** Release, in 2015<br /> **** are we ready<br /> **** do we need to release<br /> **** about 2017 we switch to time based releases, for both minor and major releases<br /> *** RMT, in 2016<br /> *** Committter selection, in 2015<br /> **** private committers list<br /> *** CoC, in 2016<br /> **** initial idea was for core to do enforcement<br /> **** now CoC does that<br /> ** One would think core doesn't have work after all that delegation<br /> ** Still lots of coordination / &quot;cat herding&quot;<br /> ** Important teams overlap with Core, CoCC is explicitly separate<br /> ** largely independent teams<br /> *** Advocacy<br /> *** Funds<br /> ** Questions:<br /> *** Stacey: Core is self selecting, plans to change that?<br /> **** Tom:<br /> ***** Not currently<br /> ***** Should definitely be long-term members for institutional memory<br /> **** Bruce: Takes a while to get there, we were looking for people a few times, didn't work out<br /> **** Robert: Self selecting is an issue, but not clear what the solution is<br /> **** Dave: It's hard to model something resilient against takeover<br /> **** Karen: At least we should be honest about this issue toward the community<br /> **** Heikki: Community is very concensus drive, hard to make decision, but also good to not piss off people. Core team has been decent at that.<br /> **** Tom: Policy that there is not more than 50% of one company<br /> **** Robert: Feels awkward to be &quot;steered&quot; by a something as opaque<br /> <br /> == Security Team - Noah ==<br /> <br /> * 17 people<br /> * receives reports of security issues<br /> * issues come from three groups<br /> ** postgres devs<br /> ** people we haven't heard of, typically something operating automated scanner<br /> *** our mailing lists are public, you know?!?<br /> ** occasionally reports by &quot;real&quot; security researchers<br /> * fixes developed by<br /> ** reports<br /> ** security team members<br /> ** &quot;outsourcing program&quot;, to non-security member committers<br /> * 7 security issues in the last year, one high risk one<br /> * We are now a CNA<br /> ** CVEs are assigned for most vulnerabilities<br /> ** new: encouragement to become our own numbering authority<br /> ** why worthwhile: CVEs assigned by outside numbering authorities by other numbering authorities<br /> ** with CNA we're supposed to be asked first<br /> ** now can update metadata for old CVEs<br /> ** also now responsible for assigning CVEs in the wider postgres space<br /> ** also get asked about filling out security questionaire<br /> * Question to audience: Problems with security model?<br /> ** Dave Cramer: Process more complicated than github's<br /> *** tarballs are released, commit messages mails are suppressed, but everyone can look at commits<br /> *** in theory we could make fix in private repo, like we did once before<br /> <br /> <br /> == PostgreSQL Community Association - Steve Singer ==<br /> <br /> * Special purpose entitity to focus on intelectual property matters<br /> * Maintain and protect assets (trademarks, domain names, not code)<br /> * does not hold copyright on postgres code, held by individuals<br /> * Canadian Not for Profit<br /> * Board:<br /> ** Dave Page<br /> ** Steve Siunger<br /> ** Jonathan Katz<br /> ** Jaime Casanova<br /> ** Peter Eisentraut<br /> ** Marc Fournier<br /> * self selecting membership, by board (past and present members)<br /> ** aware of issues around that<br /> * Why:<br /> ** ensure fair use of PostgreSQL brand assets<br /> ** otherwise somebody else can call something postgres<br /> ** requires enforcement, otherwise trademark is abandoned and can't be enforced<br /> ** keep PG fre and open source<br /> * Where:<br /> ** Canada<br /> ** US<br /> ** Europe<br /> * How:<br /> ** controlled fair use allowed<br /> ** example: Brand names involving postgres need to be licensed<br /> ** standard license for templates, but often companies want modified terms<br /> * Key Activities:<br /> ** Oppose Trademark Registrations<br /> *** some conflicts around that are public, others not<br /> ** Ask that the trademarks be used properly<br /> ** License where needed<br /> * Highlights<br /> ** Fundacion PostgreSQL conclusion (after three years and lots of money)<br /> ** Trademark oppositions, cease &amp; desists<br /> ** Donor Recognition Program<br /> * Why private?<br /> ** public statement can have legal consequences, need to be reviewed by lawyers, costs money for review<br /> ** rather work amiciably, rather than public shaming, often succeeds<br /> ** some unlicensed uses are because we didn't have the time/money to do something about them<br /> * Financial highlights:<br /> ** Net loss: CAD 11k<br /> ** Goal: Improve financials via Donor recognition program<br /> ** Various levels that grant public recognition<br /> * Question:<br /> <br /> <br /> == Code of Conduct Committee - Eliza Benett ==<br /> <br /> * Eliza is chair since 2014<br /> * Overview of the last year<br /> ** Representation:<br /> *** 5 members, 3 male identifying, 2 female, plus chair<br /> *** geographic spread: 2 from Canada, 1 from Asial, one american in Asia, 1 from Europe<br /> *** Volunteers, selected by current members<br /> *** How can group size be increased<br /> ** Activity<br /> *** 8 complaints in 2023/24 (inappropriate conduct on community channels (2), at community events (2), non-community channel (3))<br /> *** Outcomes: no violation (3), out of scope (2), corrective recommended and taken (2), corrective action recommended (1)<br /> *** processing done by lead investigator, always with at least one other person<br /> *** [couldn't keep up]<br /> ** Trends:<br /> *** 2023/24 have been busy<br /> *** Complaints on social media, [couldn't keep up]<br /> ** Opportunities:<br /> *** need new members<br /> *** update CoC policy, hasn't bene updated significantly since introduction<br /> *** improve enforcement, appeals process<br /> *** inform community events about CoC existance / use of CoC[C]<br /> ** Revisions:<br /> *** Roles &amp; responsibilities<br /> *** Scope clarification<br /> *** Appeals process<br /> *** Record sharing and retention, records exist since 2019 (unsure)<br /> *** Communication processes<br /> *** ....<br /> *** proposed timeline: Summer 2024, engegement on policy revisions, September: 2024, in effect<br /> <br /> <br /> == Port report - Thomas Munro ==<br /> <br /> * lots of platform that postgres used to run on, but doesn't anymore<br /> * still support about 11 platforms<br /> * most users on linux, some on windows, the rest probably miniscule<br /> * Robert Haas and Devrim Gunduz think there are way more on windows than Thomas guessed<br /> * Dave Page says some windows apps embed postgres<br /> * Estimate about problem per platform, overwhelmingly windows<br /> ** Robert: Lots of it just perl crap<br /> * Highlights:<br /> ** AIX was dropped in 2017, suddenly IBM folks appeared, rely on Postgres<br /> *** AIX is weird, unusual linker/object format<br /> *** Old OS versions in buildfarm<br /> *** Vendor compiler requires extra work<br /> **** some bugs got fixed<br /> ** HP-UX was dropped in 16<br /> *** no real complaints, one mention<br /> ** Solaris &lt;= 11.3 in 17 was dropped<br /> *** solaris is less painful<br /> *** user interest, solaris buildfarm animals by users<br /> *** also illumos<br /> *** they stopped increasing versions after 11.4<br /> **** should we drop support for the sunpro compiler?<br /> *** won't support newer things like direct-io<br /> ** FreeBSD:<br /> *** Short supported version window, buildfarm animals exist that are very old<br /> ** OpenBSD:<br /> *** uses libressl<br /> ** Linux+musl<br /> *** different libc, a few incompatibilities that we have to deal with<br /> *** Have a buildfarm animal now<br /> ** Windows:<br /> *** Removed old perl based build system, did make things easier<br /> *** Thomas never used windows, despite working for Microsoft<br /> *** Locale support is very instable, breaks frequently<br /> **** particularly with turkish locale, breaks all PG databases with the locale, but also some others<br /> *** truncations fail occasionally<br /> **** causes corruptions<br /> *** Socket event loss, causes weird problems<br /> *** Thomas tries to fix things, but doesn't use windows, hard to find testing<br /> *** Thomas finds our state of windows support embarassing<br /> *** We ought to make some decisions to improve the situation, requires making dropping some filesystems<br /> **** who makes decision?</div>

Andresfreund https://wiki.postgresql.org/index.php?title=CI_Providers&diff=38137 2023-08-08T02:35:56Z

<p>Andresfreund: </p> <hr /> <div>Overview of CI providers and the limits imposed<br /> <br /> {| class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ CI Providers<br /> |-<br /> ! Provider !! OSs !! Open source specific offering !! Monthly Minutes &lt;br /&gt; &quot;open repo&quot; !! Monthly Minutes&lt;br /&gt;open source !! Allows VMs !! Public Logs !! Links<br /> |-<br /> | cirrus-ci || linux, macos, windows, *bsd || no || ~ 16.6k 1 core linux&lt;br /&gt;833 4 core macos|| same || yes || yes || https://cirrus-ci.org/pricing/#compute-services &lt;br /&gt; https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/<br /> |-<br /> | github actions || linux, macos, windows || kinda || unlimited (?) || same || no || no || https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration<br /> |-<br /> | circle-ci || linux, macos, windows || yes || 400 || 80k 1 core linux&lt;br /&gt;600 4 core macos || yes (?) || no || https://circleci.com/pricing/ https://circleci.com/open-source/ &lt;br /&gt; https://circleci.com/product/features/resource-classes/<br /> |-<br /> | gitlab || linux, macos, windows || yes || 400 || if approved 50k 1 core linux&lt;br /&gt;8.3k 4 core macos || no || yes || https://about.gitlab.com/pricing/ &lt;br /&gt; https://about.gitlab.com/solutions/open-source/<br /> |-<br /> | appveyor || linux, macos, windows || yes || 1 concurrent job || 50% off price || ? || ? || https://www.appveyor.com/pricing/ &lt;br /&gt; <br /> |}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=CI_Providers&diff=38136 2023-08-08T02:32:42Z

<p>Andresfreund: </p> <hr /> <div>Overview of CI providers and the limits imposed<br /> <br /> {| class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ CI Providers<br /> |-<br /> ! Provider !! OSs !! Open source specific offering !! Monthly Minutes &lt;br /&gt; &quot;open repo&quot; !! Monthly Minutes&lt;br /&gt;open source !! Allows VMs !! Public Logs !! Links<br /> |-<br /> | cirrus-ci || macos, linux, windows, *bsd || no || ~ 16.6k 1 core linux&lt;br /&gt;833 4 core macos|| same || yes || yes || https://cirrus-ci.org/pricing/#compute-services &lt;br /&gt; https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/<br /> |-<br /> | github actions || macos, linux, windows || kinda || unlimited (?) || same || no || no || https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration<br /> |-<br /> | circle-ci || macos, linux, windows || yes || 400 || 80k 1 core linux&lt;br /&gt;600 4 core macos || yes (?) || no || https://circleci.com/pricing/ https://circleci.com/open-source/ &lt;br /&gt; https://circleci.com/product/features/resource-classes/<br /> |-<br /> | gitlab || macos, linux, windows || yes || 400 || if approved 50k 1 core linux&lt;br /&gt;8.3k 4 core macos || no || yes || https://about.gitlab.com/pricing/ &lt;br /&gt; https://about.gitlab.com/solutions/open-source/<br /> |}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=CI_Providers&diff=38135 2023-08-08T02:29:45Z

<p>Andresfreund: </p> <hr /> <div>Overview of CI providers and the limits imposed<br /> <br /> {| class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ CI Providers<br /> |-<br /> ! Provider !! OSs !! Open source specific offering !! Monthly Minutes &quot;open repo&quot; !! Monthly Minutes open source !! Allows VMs !! Public Logs !! Links<br /> |-<br /> | cirrus-ci || macos, linux, windows, *bsd || no || ~ 16.6k 1 core linux, 833 4 core macos|| same || yes || yes || https://cirrus-ci.org/pricing/#compute-services https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/<br /> |-<br /> | github actions || macos, linux, windows || kinda || unlimited (?) || same || no || no || https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration<br /> |-<br /> | circle-ci || macos, linux, windows || yes || 400 || 80k 1 core linux, 600 4 core macos || yes (?) || no || https://circleci.com/pricing/ https://circleci.com/open-source/ https://circleci.com/product/features/resource-classes/<br /> |-<br /> | gitlab || macos, linux, windows || yes || 400 || if approved 50k 1 core linux, 8.3k 4 core macos || no || yes || https://about.gitlab.com/pricing/ https://about.gitlab.com/solutions/open-source/<br /> |}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=CI_Providers&diff=38134 2023-08-08T02:27:31Z

<p>Andresfreund: </p> <hr /> <div>Overview of CI providers and the limits imposed<br /> <br /> {| class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ CI Providers<br /> |-<br /> ! Provider !! OSs !! Open source specific offering !! Monthly Minutes &quot;open repo&quot; !! Monthly Minutes open source !! Allows VMs !! Public Logs !! Links<br /> |-<br /> | cirrus-ci || macos, linux, windows, *bsd || no || ~ 16.6k 1 core linux, 833 4 core macos|| same || yes || yes || https://cirrus-ci.org/pricing/#compute-services https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/<br /> |-<br /> | github actions || macos, linux, windows || kinda || unlimited (?) || same || no || no || https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration<br /> |-<br /> | circle-ci || macos, linux, windows || yes || 400 || 80k 1 core linux, 600 4 core macos || yes (?) || ? || https://circleci.com/pricing/ https://circleci.com/open-source/ https://circleci.com/product/features/resource-classes/<br /> |-<br /> | gitlab || macos, linux, windows || yes || 400 || if approved 50k 1 core linux, 8.3k 4 core macos || no || ? || https://about.gitlab.com/pricing/ https://about.gitlab.com/solutions/open-source/<br /> |}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=CI_Providers&diff=38133 2023-08-08T02:26:43Z

<p>Andresfreund: </p> <hr /> <div>Overview of CI providers and the limits imposed<br /> <br /> {| class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ CI Providers<br /> |-<br /> ! Provider !! OSs !! Open source specific offering !! Monthly Minutes &quot;open repo&quot; !! Monthly Minutes open source !! Allows VMs !! Links<br /> |-<br /> | cirrus-ci || macos, linux, windows, *bsd || no || ~ 16.6k 1 core linux, 833 4 core macos|| same || yes || https://cirrus-ci.org/pricing/#compute-services https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/<br /> |-<br /> | github actions || macos, linux, windows || kinda || unlimited (?) || same || no || https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration<br /> |-<br /> | circle-ci || macos, linux, windows || yes || 400 || 80k 1 core linux, 600 4 core macos || yes (?) || https://circleci.com/pricing/ https://circleci.com/open-source/ https://circleci.com/product/features/resource-classes/<br /> |-<br /> | gitlab || macos, linux, windows || yes || 400 || if approved 50k 1 core linux, 8.3k 4 core macos || no || https://about.gitlab.com/pricing/ https://about.gitlab.com/solutions/open-source/<br /> |}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=CI_Providers&diff=38132 2023-08-08T01:56:34Z

<p>Andresfreund: Created page with &quot;Overview of CI providers and the limits imposed {| class=&quot;wikitable&quot; style=&quot;margin:auto&quot; |+ CI Providers |- ! Provider !! OSs !! Open source specific offering !! Monthly Minu...&quot;</p> <hr /> <div>Overview of CI providers and the limits imposed<br /> <br /> {| class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ CI Providers<br /> |-<br /> ! Provider !! OSs !! Open source specific offering !! Monthly Minutes &quot;open repo&quot; !! Monthly Minutes open source !! Allows VMs !! Links<br /> |-<br /> | cirrus-ci || macos, linux, windows, *bsd || no || ~ 16.6k linux 1 core linux, 833 4 core macos|| same || yes || https://cirrus-ci.org/pricing/#compute-services https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/<br /> |-<br /> | github actions || macos, linux, windows || kinda || unlimited (?) || same || no || https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration<br /> |-<br /> | circle-ci || macos, linux, windows || yes || 400 || 80k 1 core linux, 600 4 core macos || yes (?) || https://circleci.com/pricing/ https://circleci.com/open-source/ https://circleci.com/product/features/resource-classes/<br /> |-<br /> | gitlab || macos, linux, windows || yes || 400 || if approved 50k 1 core linux, 8.3k 4 core macos || no || https://about.gitlab.com/pricing/ https://about.gitlab.com/solutions/open-source/<br /> |}</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_16_Open_Items&diff=38075 2023-07-25T03:33:47Z

<p>Andresfreund: </p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * {{messageLink|CAJKUy5g2uZRrUDZJ8p-%3DgiwcSHVUn0c9nmdxPSY0jF0Ov8VoEA@mail.gmail.com|Assertion failure !bms_overlap(joinrel-&gt;relids, required_outer)}}<br /> ** Owner: Tom Lane<br /> * {{messageLink|CAD21AoDvDmUQeJtZrau1ovnT_smN940%3DKp6mszNGK3bq9yRN6g%40mail.gmail.com| Performance degradation on concurrent COPY into a single relation in PG16 }}<br /> ** Owner: Andres Freund<br /> * {{messageLink|36a6e89689716c2ca1fae8adc8e84601a041121c.camel@j-davis.com| Unexplained behavior when ICU rules is the empty string.}}<br /> ** Owner: Peter Eisentraut<br /> * {{messageLink|ZKy4AdrLEfbqrxGJ@telsasoft.com|REINDEX segv on null pointer in RemoveFromWaitQueue}}<br /> ** Owner: Andres Freund<br /> ** {{messageLink|CAD21AoDs7vzK7NErse7xTruqY-FXmM%2B3K00SdBtMcQhiRNkoeQ%40mail.gmail.com|Patch}}<br /> <br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * Add any items that need discussion here..<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br /> ** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br /> ** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br /> ** Unclear what the user impact is.<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br /> ** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer] [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report] [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report] [https://www.postgresql.org/message-id/flat/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA%40mail.gmail.com Another report] [https://www.postgresql.org/message-id/flat/a53cacb0-8835-57d6-31e4-4c5ef196de1a@deepbluecap.com Another report]<br /> <br /> * [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br /> ** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/ZArVOMifjzE7f8W7%40paquier.xyz Requiring recovery.signal or standby.signal when recovering with a backup_label]<br /> ** This is a rather old behavior that affects all stable branches, still not something that should be backpatched as-is.<br /> <br /> * {{messageLink|cfcca574-6967-c5ab-7dc3-2c82b6723b99@mail.ru|pg_visibility's pg_check_visible() yields false positive when working in parallel with autovacuum}}<br /> ** {{messageLink|1649062270.289865713@f403.i.mail.ru|Thread with patch}} [https://commitfest.postgresql.org/43/3739/ CF Entry]<br /> <br /> * {{messageLink|1516594.1681482708@sss.pgh.pa.us|We are not compatible with newly-released LLVM 16}}<br /> ** {{messageLink|CA%2BhUKGKNX_%3Df%2B1C4r06WETKTq0G4Z_7q4L4Fxn5WWpMycDj9Fw%40mail.gmail.com|Patch}}<br /> ** Owner: Thomas Munro (volunteer LLVM API change chaser)<br /> <br /> <br /> === Fixed issues ===<br /> <br /> * [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and &gt;MaxAllocSize lengths]<br /> ** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br /> ** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br /> ** Affects all stable branches.<br /> ** Fixed at: {{PgCommitURL|8fcb32db98eda1ad2a0c0b40b1cbb5d9a7aa68f0}} and {{PgCommitURL|ffd1b6bb6f8a2ffc929699772610c6925364dbb3}} for HEAD.<br /> <br /> * [https://www.postgresql.org/message-id/flat/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Fix fseek() detection of unseekable files on WIN32]<br /> ** Fixed at: {{PgCommitURL|a923e21631a29dc8b8781d7d02b5003d0df64ca3}} and {{PgCommitURL|765f5df726918bcdcfd16bcc5418e48663d1dd59}}, down to 14.<br /> <br /> * {{messageLink|CAAKRu_bETD%2BAri600h6fRjX2p8rJSeMAUp%3D_y88juqOZgouTSg%40mail.gmail.com|Can't disable autovacuum cost delay through storage parameter}}<br /> ** Fixed at: {{PgCommitURL|bfac8f8bc4a44c67c9f35b5266676278e4ba1217}}, down to 11.<br /> <br /> * {{messageLink|CAJ7c6TMBTN3rcz4%3DAjYhLPD_w3FFT0Wq_C15jxCDn8U4tZnH1g@mail.gmail.com| EPQ misbehaves for inherited/partitioned tables}}<br /> ** Fixed at: {{PgCommitURL|70b42f279}}, down to 14.<br /> <br /> * {{messageLink|ZEZDj1H61ryrmY9o@msg.df7cb.de|could not extend file &quot;base/5/3501&quot; with FileFallocate(): Interrupted system call}}<br /> ** Original commit: {{PgCommitURL|4d330a61bb1}}<br /> ** Fixed at: {{PgCommitURL|0d369ac650}}<br /> <br /> * {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br /> ** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|&quot;PANIC: could not open critical system index 2662&quot; - twice}}<br /> ** Fixed at: {{PgCommitURL|c66a7d75}}, down to 11.<br /> <br /> * {{messageLink|17997-a044c27aef95daf8@postgresql.org|Assertion failure when attaching a partition index}}<br /> ** Fixed at: {{PgCommitURL|38ea6aa9}}, down to 11.<br /> ** Fixed at: {{PgCommitURL|cfc43aeb}}, down to 11.<br /> <br /> * {{messageLink|ae46f2fb-5586-3de0-b54b-1bb0f6410ebd@inbox.ru|Issues with calculations of LimitAdditionalPins}}<br /> ** Fixed at: {{PgCommitURL|bd2f46c6559}}<br /> <br /> == Non-bugs ==<br /> <br /> * {{messageLink|17862-1ab8f74b0f7b0611@postgresql.org|WindowAgg startup costs don't take into account partition bound. Can lead to incorrect use of cheap startup plans}}<br /> ** {{messageLink|CAApHDvrB0S5BMv+0-wTTqWFE-BJ0noWqTnDu9QQfjZ2VSpLv_g@mail.gmail.com|Patch to fix and discussion}}<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 16beta3 ===<br /> * {{messageLink|ZJp921+nITFnvBVS@paquier.xyz|Add TLI number to name of files generated by pg_waldump --save-fullpage}}<br /> ** Owner: Michael Paquier<br /> ** Fixed at: {{PgCommitURL|b381d9637030c163c3b1f8a9d3de51dfc1b4ee58}}<br /> * {{messageLink|e587e2ee-7de0-88a2-10f8-c7cf001bab8c%40postgrespro.ru|psql: Add role's membership options to the \du+ command}}<br /> ** Fixed at: {{PgCommitURL|0a1d2a7df852f16c452eef8a83003943125162c7}}<br /> <br /> === resolved before 16beta2 ===<br /> * {{messageLink|17978-12f3d93a55297266@postgresql.org|wrong join order subsequent to removal of delay_upper_joins check}}<br /> ** Owner: Tom Lane<br /> ** Fixed at: {{PgCommitURL|3af87736bf5d5c7bea086d962afc2bbf4f29279a}}<br /> * {{messageLink|DFBB2D25-DE97-49CA-A60E-07C881EA59A7@winand.at|Inconsistent nulling bitmap in nestloop parameters}}<br /> ** Owner: Tom Lane<br /> ** All known issues fixed as of {{PgCommitURL|efeb12ef0bfef0b5aa966a56bb4dbb1f936bda0c}}<br /> * {{messageLink|17976-4b638b525e9a983b@postgresql.org|join removal can no longer skip updating EquivalenceClasses}}<br /> ** Owner: Tom Lane<br /> ** Fixed at: {{PgCommitURL|f4c00d138f6dea4c9d8af8ec280b7edc9b0a29e1}}<br /> * {{messageLink|CAH2-Wz%3D8Z9qY58bjm_7TAHgtW6RzZ5Ke62q5emdCEy9BAzwhmg%40mail.gmail.com|Cleaning up nbtree after logical decoding on standby work}}<br /> ** Owner: Peter Geoghegan, Andres Freund<br /> ** Original commit: {{PgCommitURL|61b313e4}}<br /> ** Fixed at: {{PgCommitURL|d088ba5a}}<br /> * {{messageLink|CAMbWs4_tuVn9EwwMcggGiZJWWstdXX_ci8FeEU17vs+4nLgw3w@mail.gmail.com|Assert failure and wrong query results due to incorrectly removing PHV}}<br /> ** Owner: Tom Lane<br /> ** Fixed at: {{PgCommitURL|9a2dbc614e6e47da3c49daacec106da32eba9467}}<br /> * {{messageLink|CAMbWs4-_vwkBij4XOQ5ukxUvLgwTm0kS5_DO9CicUeKbEfKjUw%40mail.gmail.com|Assert failure of the cross-check for nullingrels}}<br /> ** Owner: Tom Lane<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** [https://commitfest.postgresql.org/43/4250/ CF Entry]<br /> ** Fixed at: {{PgCommitURL|991a3df22}}<br /> * Switch to ICU for 17?<br /> ** Owner: Jeff Davis<br /> ** {{messageLink|82c4c816-06f6-d3e3-ba02-fca4a5cef065@enterprisedb.com|I suggest waiting until next week to commit it and then see what happens}}<br /> ** [https://commitfest.postgresql.org/42/4169/ CF Entry]<br /> ** Open item description not clear; if it is an open item, it's redundant with the issue &quot;The rules for choosing default ICU locale seem pretty unfriendly&quot;. Closed. <br /> * {{messageLink|874jp9f5jo.fsf@news-spur.riddles.org.uk|The rules for choosing default ICU locale seem pretty unfriendly}}<br /> ** Owner: Jeff Davis<br /> ** Fixed at: {{PgCommitURL|2535c74b1a}}, {{PgCommitURL|f3a01af29b}}<br /> * {{messageLink|20230613211246.GA219055@nathanxps13|ff9618e creates cache lookup hazards with partition trees}}<br /> ** Owner: Nathan Bossart, &lt;s&gt;Jeff Davis&lt;/s&gt;<br /> ** Fixed at: {{PgCommitURL|4dbdb82513}}, {{PgCommitURL|c2122aae63}}<br /> === resolved before 16beta1 ===<br /> * {{messageLink|CAHewXNnu7u1aT%3D%3DWjnCRa%2BSzKb6s80hvwPP_9eMvvvtdyFdqjw%40mail.gmail.com|ERROR: wrong varnullingrels (b 5 7) (expected (b)) for Var 3/3}}<br /> ** Fixed at: {{PgCommitURL|d0f952691}}<br /> * {{messageLink|d46f9265-ff3c-6743-2278-6772598233c2%40pgmasters.net|Possible regression setting GUCs on \connect}}<br /> ** Owner: Alexander Korotkov<br /> ** Discussion on reverting {{PgCommitURL|096dd80f3}}<br /> ** Original commit: {{PgCommitURL|096dd80f3}}<br /> ** Reverted at: {{PgCommitURL|b9a7a822723aebb16cbe7e5fb874e5124745b07e}}<br /> <br /> * Planner makes improper clause pushdown decisions due to outer-join-aware-Vars changes<br /> ** {{messageLink|0b819232-4b50-f245-1c7d-c8c61bf41827@postgrespro.ru|Clause accidentally pushed down}}<br /> ** {{messageLink|CAHewXNks3w_Vy9CWoVtHx1XSaeiFpsOzh-zy5eu0Khp1PtG1sA@mail.gmail.com|wrong results due to qual pushdown}}<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** Fixed at: {{PgCommitURL|9df8f903eb6758be5a19e66cdf77e922e9329c31}}<br /> <br /> * Revert {{PgCommitURL|ec386948948}}, per {{messageLink|20230330105325.y6uvpalspynf2frt@alvherre.pgsql|Re: &quot;variable not found in subplan target list&quot;}}<br /> ** Reverted at {{PgCommitURL|5472743d9e8}}<br /> <br /> * [https://www.postgresql.org/message-id/CAEZATCWETioXs5kY8vT6BVguY41_wD962VDk%3Du_Nvd7S1UXzuQ%40mail.gmail.com ERROR: ORDER/GROUP BY expression not found in targetlist]<br /> ** Fixed at: {{PgCommitURL|da5800d5fa636c6e10c9c98402d872c76aa1c8d0}}<br /> <br /> * [https://www.postgresql.org/message-id/20230212233711.GA1316@telsasoft.com various elogs hit by sqlsmith (ExecRTCheckPerms() and many prunable partitions)]<br /> ** Fixed at: {{PgCommitURL|c7468c73f7b6e842a53c12eaee5578a76a8fa7a6}}<br /> <br /> * [https://www.postgresql.org/message-id/20230228235834.GC30529@telsasoft.com pg_dump: zlib compression fails for empty objects (LOs)]<br /> ** Fixed at: {{PgCommitURL|00d9dcf5bebbb355152a60f0e2120cdf7f9e7ddd}}<br /> <br /> * [https://www.postgresql.org/message-id/20230227044910.GO1653@telsasoft.com pg_dump: lz4 compression uses no persistent state and writes a block header for every row]<br /> ** Fixed at: {{PgCommitURL|0070b66fef21e909adb283f7faa7b1978836ad75}}<br /> <br /> * {{messageLink|3590249.1680971629@sss.pgh.pa.us|Assertion failure with parallel full hash join}}<br /> ** Fixed at: {{PgCommitURL|b37d051b0e59e4324e346655a27509507813db79}}<br /> <br /> * {{messageLink|ZDDO6jaESKaBgej0@tamriel.snowman.net|De-revert &quot;Add support for Kerberos credential delegation&quot;}}<br /> ** Owner: Stephen Frost<br /> ** Original commit: {{PgCommitURL|3d4fa227bce4294ce1cc214b4a9d3b7caa3f0454}}<br /> ** Revert: ({{PgCommitURL|3d03b24c350ab060bb223623bdff38835bd7afd0}}<br /> ** De-Revert: {{PgCommitURL|6633cfb21691840c33816a6dacaca0b504efb895}}<br /> ** Resolved at: {{PgCommitURL|f7431bca8b0138bdbce7025871560d39119565a0}}<br /> <br /> * {{messageLink|c39be3c5-c1a5-1e33-1024-16f527e251a4@enterprisedb.com|SSL tests break on non-existing system CA pool}}<br /> ** Fixed at: {{PgCommitURL|0b5d1fb36adda612bd3d5d032463a6eeb0729237}}<br /> <br /> * {{messageLink|CAD21AoBS7o6Ljt_vfqPQPf67AhzKu3fR0iqk8B%3DvVYczMugKMQ%40mail.gmail.com|VacuumUpdateCosts() logging condition incorrect for some initial values of vacuum_cost_delay}}<br /> ** Fixed at: {{PgCommitURL|a9781ae11ba2fdb44a3a72c9a7ebb727140b25c5}}<br /> <br /> * {{messageLink|CA%2BhUKGJ-ZPJwKHVLbqye92-ZXeLoCHu5wJL6L6HhNP7FkJ%3DmeA%40mail.gmail.com|check_strxfrm_bug()}}<br /> ** Owner: Thomas Munro<br /> ** Fixed at: {{PgCommitURL|7d3d72b55edd1b7552a9a358991555994efab0e9}}<br /> <br /> * {{messageLink|20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de|Should we remove vacuum_defer_cleanup_age?}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|1118cd37eb61e6a2428f457a8b2026a7bb3f801a}}<br /> <br /> * {{messageLink|2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com|Add two missing tests in 035_standby_logical_decoding.pl}}<br /> ** Fixed at: {{PgCommitURL|376dc820531bafcbf105fff74c5b14c23d9950af}}<br /> ** Fixed at: {{PgCommitURL|a6e04b1d20c2e9cece9b64bb5b36ebfdc3a9031b}}<br /> <br /> * {{messageLink|b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com|Performance regression due to SQLValueFunction removal}}<br /> ** Fixed at: {{PgCommitURL|d8c3106bb60e4f87be595f241e173ba3c2b7aa2c}}<br /> <br /> * {{messageLink|20230419172326.dhgyo4wrrhulovt6%40awork3.anarazel.de|pg_stat_io not tracking smgrwriteback() is confusing}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|093e5c57d506783a95dd8feddd9a3f2651e1aeba}}<br /> <br /> * {{messageLink|ZFhCyn4Gm2eu60rB@paquier.xyz|Table data compression is broken with pg_dump --compress lz4}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|1a05c1d252993b0a59c58a6daf91a2df9333044f}}<br /> <br /> * {{messageLink|94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com|Valgrind unhappy with LZ4F code in pg_dump}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|3c18d90f8907e53c3021fca13ad046133c480e4d}}<br /> <br /> * {{messageLink|20230509190247.3rrplhdgem6su6cg@awork3.anarazel.de|walsender performance regression due to logical decoding on standby changes}}<br /> ** Owner: Andres Freund<br /> ** Original commit: {{PgCommitURL|e101dfac}}<br /> ** Fixed at: {{PgCommitURL|bc971f4025c378ce500d86597c34b0ef996d4d8c}}<br /> <br /> == Won't Fix ==<br /> <br /> * Is it OK that WL_SOCKET_ACCEPT is less fair on Windows than on Unix (and than the coding before 16) when there are multiple server sockets configured?<br /> ** {{messageLink|CA%2BhUKG%2BA2dk29hr5zRP3HVJQ-_PncNJM6HVQ7aaYLXLRBZU-xw%40mail.gmail.com|WL_SOCKET_ACCEPT fairness on Windows}} has a (blind) patch to fix that, but would need a Windows hacker to test<br /> ** Owner: Thomas Munro<br /> ** Original commit: {{PgCommitURL|7389aad6}}<br /> ** Issue reclassified as a non-critical improvement to be [https://commitfest.postgresql.org/43/4263/ considered for 17]<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Beta 3: (August 10, 2023)<br /> * Beta 2: June 29, 2023<br /> * Beta 1: May 25, 2023<br /> * Feature Freeze: April 8, 2023 0:00 AoE ('''Last Day to Commit Features''')<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 15 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_16_Open_Items&diff=38000 2023-06-21T03:49:13Z

<p>Andresfreund: </p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Switch to ICU for 17?<br /> ** Owner: Jeff Davis<br /> ** {{messageLink|82c4c816-06f6-d3e3-ba02-fca4a5cef065@enterprisedb.com|I suggest waiting until next week to commit it and then see what happens}}<br /> ** [https://commitfest.postgresql.org/42/4169/ CF Entry]<br /> * {{messageLink|e587e2ee-7de0-88a2-10f8-c7cf001bab8c%40postgrespro.ru|psql: Add role's membership options to the \du+ command}}<br /> ** [https://commitfest.postgresql.org/43/4116/ CF Entry]<br /> ** NOTE: This is not a committed feature for v16<br /> * {{messageLink|874jp9f5jo.fsf@news-spur.riddles.org.uk|The rules for choosing default ICU locale seem pretty unfriendly}}<br /> ** Owner: Jeff Davis<br /> * {{messageLink|20230613211246.GA219055@nathanxps13|ff9618e creates cache lookup hazards with partition trees}}<br /> ** Owner: Nathan Bossart, &lt;s&gt;Jeff Davis&lt;/s&gt;<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * [https://www.postgresql.org/message-id/268fd337-8bb7-92e6-0da2-416c022c11f3%40enterprisedb.com Reconsider a utility_query_id GUC to control if query jumbling of utilities can go through the past string-only mode and the new mode?]<br /> ** Potential owner: Michael Paquier<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br /> ** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br /> ** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br /> ** Unclear what the user impact is.<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br /> ** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer] [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report] [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report] [https://www.postgresql.org/message-id/flat/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA%40mail.gmail.com Another report] [https://www.postgresql.org/message-id/flat/a53cacb0-8835-57d6-31e4-4c5ef196de1a@deepbluecap.com Another report]<br /> <br /> * [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br /> ** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/ZArVOMifjzE7f8W7%40paquier.xyz Requiring recovery.signal or standby.signal when recovering with a backup_label]<br /> ** This is a rather old behavior that affects all stable branches, still not something that should be backpatched as-is.<br /> <br /> * {{messageLink|cfcca574-6967-c5ab-7dc3-2c82b6723b99@mail.ru|pg_visibility's pg_check_visible() yields false positive when working in parallel with autovacuum}}<br /> ** {{messageLink|1649062270.289865713@f403.i.mail.ru|Thread with patch}} [https://commitfest.postgresql.org/43/3739/ CF Entry]<br /> <br /> * {{messageLink|1516594.1681482708@sss.pgh.pa.us|We are not compatible with newly-released LLVM 16}}<br /> ** {{messageLink|CA%2BhUKGKNX_%3Df%2B1C4r06WETKTq0G4Z_7q4L4Fxn5WWpMycDj9Fw%40mail.gmail.com|Patch}}<br /> ** Owner: Thomas Munro (volunteer LLVM API change chaser)<br /> <br /> * {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br /> ** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|&quot;PANIC: could not open critical system index 2662&quot; - twice}}<br /> <br /> === Fixed issues ===<br /> <br /> * [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and &gt;MaxAllocSize lengths]<br /> ** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br /> ** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br /> ** Affects all stable branches.<br /> ** Fixed at: {{PgCommitURL|8fcb32db98eda1ad2a0c0b40b1cbb5d9a7aa68f0}} and {{PgCommitURL|ffd1b6bb6f8a2ffc929699772610c6925364dbb3}} for HEAD.<br /> <br /> * [https://www.postgresql.org/message-id/flat/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Fix fseek() detection of unseekable files on WIN32]<br /> ** Fixed at: {{PgCommitURL|a923e21631a29dc8b8781d7d02b5003d0df64ca3}} and {{PgCommitURL|765f5df726918bcdcfd16bcc5418e48663d1dd59}}, down to 14.<br /> <br /> * {{messageLink|CAAKRu_bETD%2BAri600h6fRjX2p8rJSeMAUp%3D_y88juqOZgouTSg%40mail.gmail.com|Can't disable autovacuum cost delay through storage parameter}}<br /> ** Fixed at: {{PgCommitURL|bfac8f8bc4a44c67c9f35b5266676278e4ba1217}}, down to 11.<br /> <br /> * {{messageLink|CAJ7c6TMBTN3rcz4%3DAjYhLPD_w3FFT0Wq_C15jxCDn8U4tZnH1g@mail.gmail.com| EPQ misbehaves for inherited/partitioned tables}}<br /> ** Fixed at: {{PgCommitURL|70b42f279}}, down to 14.<br /> <br /> * {{messageLink|ZEZDj1H61ryrmY9o@msg.df7cb.de|could not extend file &quot;base/5/3501&quot; with FileFallocate(): Interrupted system call}}<br /> ** Original commit: {{PgCommitURL|4d330a61bb1}}<br /> ** Fixed at: {{PgCommitURL|0d369ac650}}<br /> <br /> == Non-bugs ==<br /> <br /> * {{messageLink|17862-1ab8f74b0f7b0611@postgresql.org|WindowAgg startup costs don't take into account partition bound. Can lead to incorrect use of cheap startup plans}}<br /> ** {{messageLink|CAApHDvrB0S5BMv+0-wTTqWFE-BJ0noWqTnDu9QQfjZ2VSpLv_g@mail.gmail.com|Patch to fix and discussion}}<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 16beta2 ===<br /> * {{messageLink|17978-12f3d93a55297266@postgresql.org|wrong join order subsequent to removal of delay_upper_joins check}}<br /> ** Owner: Tom Lane<br /> ** Fixed at: {{PgCommitURL|3af87736bf5d5c7bea086d962afc2bbf4f29279a}}<br /> * {{messageLink|DFBB2D25-DE97-49CA-A60E-07C881EA59A7@winand.at|Inconsistent nulling bitmap in nestloop parameters}}<br /> ** Owner: Tom Lane<br /> ** All known issues fixed as of {{PgCommitURL|efeb12ef0bfef0b5aa966a56bb4dbb1f936bda0c}}<br /> * {{messageLink|17976-4b638b525e9a983b@postgresql.org|join removal can no longer skip updating EquivalenceClasses}}<br /> ** Owner: Tom Lane<br /> ** Fixed at: {{PgCommitURL|f4c00d138f6dea4c9d8af8ec280b7edc9b0a29e1}}<br /> * {{messageLink|CAH2-Wz%3D8Z9qY58bjm_7TAHgtW6RzZ5Ke62q5emdCEy9BAzwhmg%40mail.gmail.com|Cleaning up nbtree after logical decoding on standby work}}<br /> ** Owner: Peter Geoghegan, Andres Freund<br /> ** Original commit: {{PgCommitURL|61b313e4}}<br /> ** Fixed at: {{PgCommitURL|d088ba5a}}<br /> * {{messageLink|CAMbWs4_tuVn9EwwMcggGiZJWWstdXX_ci8FeEU17vs+4nLgw3w@mail.gmail.com|Assert failure and wrong query results due to incorrectly removing PHV}}<br /> ** Owner: Tom Lane<br /> ** Fixed at: {{PgCommitURL|9a2dbc614e6e47da3c49daacec106da32eba9467}}<br /> * {{messageLink|CAMbWs4-_vwkBij4XOQ5ukxUvLgwTm0kS5_DO9CicUeKbEfKjUw%40mail.gmail.com|Assert failure of the cross-check for nullingrels}}<br /> ** Owner: Tom Lane<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** [https://commitfest.postgresql.org/43/4250/ CF Entry]<br /> ** Fixed at: {{PgCommitURL|991a3df22}}<br /> <br /> === resolved before 16beta1 ===<br /> * {{messageLink|CAHewXNnu7u1aT%3D%3DWjnCRa%2BSzKb6s80hvwPP_9eMvvvtdyFdqjw%40mail.gmail.com|ERROR: wrong varnullingrels (b 5 7) (expected (b)) for Var 3/3}}<br /> ** Fixed at: {{PgCommitURL|d0f952691}}<br /> * {{messageLink|d46f9265-ff3c-6743-2278-6772598233c2%40pgmasters.net|Possible regression setting GUCs on \connect}}<br /> ** Owner: Alexander Korotkov<br /> ** Discussion on reverting {{PgCommitURL|096dd80f3}}<br /> ** Original commit: {{PgCommitURL|096dd80f3}}<br /> ** Reverted at: {{PgCommitURL|b9a7a822723aebb16cbe7e5fb874e5124745b07e}}<br /> <br /> * Planner makes improper clause pushdown decisions due to outer-join-aware-Vars changes<br /> ** {{messageLink|0b819232-4b50-f245-1c7d-c8c61bf41827@postgrespro.ru|Clause accidentally pushed down}}<br /> ** {{messageLink|CAHewXNks3w_Vy9CWoVtHx1XSaeiFpsOzh-zy5eu0Khp1PtG1sA@mail.gmail.com|wrong results due to qual pushdown}}<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** Fixed at: {{PgCommitURL|9df8f903eb6758be5a19e66cdf77e922e9329c31}}<br /> <br /> * Revert {{PgCommitURL|ec386948948}}, per {{messageLink|20230330105325.y6uvpalspynf2frt@alvherre.pgsql|Re: &quot;variable not found in subplan target list&quot;}}<br /> ** Reverted at {{PgCommitURL|5472743d9e8}}<br /> <br /> * [https://www.postgresql.org/message-id/CAEZATCWETioXs5kY8vT6BVguY41_wD962VDk%3Du_Nvd7S1UXzuQ%40mail.gmail.com ERROR: ORDER/GROUP BY expression not found in targetlist]<br /> ** Fixed at: {{PgCommitURL|da5800d5fa636c6e10c9c98402d872c76aa1c8d0}}<br /> <br /> * [https://www.postgresql.org/message-id/20230212233711.GA1316@telsasoft.com various elogs hit by sqlsmith (ExecRTCheckPerms() and many prunable partitions)]<br /> ** Fixed at: {{PgCommitURL|c7468c73f7b6e842a53c12eaee5578a76a8fa7a6}}<br /> <br /> * [https://www.postgresql.org/message-id/20230228235834.GC30529@telsasoft.com pg_dump: zlib compression fails for empty objects (LOs)]<br /> ** Fixed at: {{PgCommitURL|00d9dcf5bebbb355152a60f0e2120cdf7f9e7ddd}}<br /> <br /> * [https://www.postgresql.org/message-id/20230227044910.GO1653@telsasoft.com pg_dump: lz4 compression uses no persistent state and writes a block header for every row]<br /> ** Fixed at: {{PgCommitURL|0070b66fef21e909adb283f7faa7b1978836ad75}}<br /> <br /> * {{messageLink|3590249.1680971629@sss.pgh.pa.us|Assertion failure with parallel full hash join}}<br /> ** Fixed at: {{PgCommitURL|b37d051b0e59e4324e346655a27509507813db79}}<br /> <br /> * {{messageLink|ZDDO6jaESKaBgej0@tamriel.snowman.net|De-revert &quot;Add support for Kerberos credential delegation&quot;}}<br /> ** Owner: Stephen Frost<br /> ** Original commit: {{PgCommitURL|3d4fa227bce4294ce1cc214b4a9d3b7caa3f0454}}<br /> ** Revert: ({{PgCommitURL|3d03b24c350ab060bb223623bdff38835bd7afd0}}<br /> ** De-Revert: {{PgCommitURL|6633cfb21691840c33816a6dacaca0b504efb895}}<br /> ** Resolved at: {{PgCommitURL|f7431bca8b0138bdbce7025871560d39119565a0}}<br /> <br /> * {{messageLink|c39be3c5-c1a5-1e33-1024-16f527e251a4@enterprisedb.com|SSL tests break on non-existing system CA pool}}<br /> ** Fixed at: {{PgCommitURL|0b5d1fb36adda612bd3d5d032463a6eeb0729237}}<br /> <br /> * {{messageLink|CAD21AoBS7o6Ljt_vfqPQPf67AhzKu3fR0iqk8B%3DvVYczMugKMQ%40mail.gmail.com|VacuumUpdateCosts() logging condition incorrect for some initial values of vacuum_cost_delay}}<br /> ** Fixed at: {{PgCommitURL|a9781ae11ba2fdb44a3a72c9a7ebb727140b25c5}}<br /> <br /> * {{messageLink|CA%2BhUKGJ-ZPJwKHVLbqye92-ZXeLoCHu5wJL6L6HhNP7FkJ%3DmeA%40mail.gmail.com|check_strxfrm_bug()}}<br /> ** Owner: Thomas Munro<br /> ** Fixed at: {{PgCommitURL|7d3d72b55edd1b7552a9a358991555994efab0e9}}<br /> <br /> * {{messageLink|20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de|Should we remove vacuum_defer_cleanup_age?}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|1118cd37eb61e6a2428f457a8b2026a7bb3f801a}}<br /> <br /> * {{messageLink|2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com|Add two missing tests in 035_standby_logical_decoding.pl}}<br /> ** Fixed at: {{PgCommitURL|376dc820531bafcbf105fff74c5b14c23d9950af}}<br /> ** Fixed at: {{PgCommitURL|a6e04b1d20c2e9cece9b64bb5b36ebfdc3a9031b}}<br /> <br /> * {{messageLink|b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com|Performance regression due to SQLValueFunction removal}}<br /> ** Fixed at: {{PgCommitURL|d8c3106bb60e4f87be595f241e173ba3c2b7aa2c}}<br /> <br /> * {{messageLink|20230419172326.dhgyo4wrrhulovt6%40awork3.anarazel.de|pg_stat_io not tracking smgrwriteback() is confusing}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|093e5c57d506783a95dd8feddd9a3f2651e1aeba}}<br /> <br /> * {{messageLink|ZFhCyn4Gm2eu60rB@paquier.xyz|Table data compression is broken with pg_dump --compress lz4}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|1a05c1d252993b0a59c58a6daf91a2df9333044f}}<br /> <br /> * {{messageLink|94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com|Valgrind unhappy with LZ4F code in pg_dump}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|3c18d90f8907e53c3021fca13ad046133c480e4d}}<br /> <br /> * {{messageLink|20230509190247.3rrplhdgem6su6cg@awork3.anarazel.de|walsender performance regression due to logical decoding on standby changes}}<br /> ** Owner: Andres Freund<br /> ** Original commit: {{PgCommitURL|e101dfac}}<br /> ** Fixed at: {{PgCommitURL|bc971f4025c378ce500d86597c34b0ef996d4d8c}}<br /> <br /> == Won't Fix ==<br /> <br /> * Is it OK that WL_SOCKET_ACCEPT is less fair on Windows than on Unix (and than the coding before 16) when there are multiple server sockets configured?<br /> ** {{messageLink|CA%2BhUKG%2BA2dk29hr5zRP3HVJQ-_PncNJM6HVQ7aaYLXLRBZU-xw%40mail.gmail.com|WL_SOCKET_ACCEPT fairness on Windows}} has a (blind) patch to fix that, but would need a Windows hacker to test<br /> ** Owner: Thomas Munro<br /> ** Original commit: {{PgCommitURL|7389aad6}}<br /> ** Issue reclassified as a non-critical improvement to be [https://commitfest.postgresql.org/43/4263/ considered for 17]<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Beta 2: June 29, 2023<br /> * Beta 1: May 25, 2023<br /> * Feature Freeze: April 8, 2023 0:00 AoE ('''Last Day to Commit Features''')<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 15 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_16_Open_Items&diff=37977 2023-06-10T04:36:50Z

<p>Andresfreund: /* Open Issues */</p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Switch to ICU for 17?<br /> ** Owner: Jeff Davis<br /> ** {{messageLink|82c4c816-06f6-d3e3-ba02-fca4a5cef065@enterprisedb.com|I suggest waiting until next week to commit it and then see what happens}}<br /> ** [https://commitfest.postgresql.org/42/4169/ CF Entry]<br /> * {{messageLink|e587e2ee-7de0-88a2-10f8-c7cf001bab8c%40postgrespro.ru|psql: Add role's membership options to the \du+ command}}<br /> ** [https://commitfest.postgresql.org/43/4116/ CF Entry]<br /> ** NOTE: This is not a committed feature for v16<br /> * {{messageLink|874jp9f5jo.fsf@news-spur.riddles.org.uk|The rules for choosing default ICU locale seem pretty unfriendly}}<br /> ** Owner: Jeff Davis<br /> * {{messageLink|ZEZDj1H61ryrmY9o@msg.df7cb.de|could not extend file &quot;base/5/3501&quot; with FileFallocate(): Interrupted system call}}<br /> ** Owner: Andres Freund<br /> ** Original commit: {{PgCommitURL|4d330a61bb1}}<br /> * {{messageLink|CAH2-Wz%3D8Z9qY58bjm_7TAHgtW6RzZ5Ke62q5emdCEy9BAzwhmg%40mail.gmail.com|Cleaning up nbtree after logical decoding on standby work}}<br /> ** Owner: Peter Geoghegan, Andres Freund<br /> ** Original commit: {{PgCommitURL|61b313e4}}<br /> * {{messageLink|DFBB2D25-DE97-49CA-A60E-07C881EA59A7@winand.at|Inconsistent nulling bitmap in nestloop parameters}}<br /> ** Owner: Tom Lane<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * [https://www.postgresql.org/message-id/268fd337-8bb7-92e6-0da2-416c022c11f3%40enterprisedb.com Reconsider a utility_query_id GUC to control if query jumbling of utilities can go through the past string-only mode and the new mode?]<br /> ** Potential owner: Michael Paquier<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br /> ** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br /> ** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br /> ** Unclear what the user impact is.<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br /> ** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer] [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report] [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report] [https://www.postgresql.org/message-id/flat/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA%40mail.gmail.com Another report] [https://www.postgresql.org/message-id/flat/a53cacb0-8835-57d6-31e4-4c5ef196de1a@deepbluecap.com Another report]<br /> <br /> * [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br /> ** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/ZArVOMifjzE7f8W7%40paquier.xyz Requiring recovery.signal or standby.signal when recovering with a backup_label]<br /> ** This is a rather old behavior that affects all stable branches, still not something that should be backpatched as-is.<br /> <br /> * {{messageLink|cfcca574-6967-c5ab-7dc3-2c82b6723b99@mail.ru|pg_visibility's pg_check_visible() yields false positive when working in parallel with autovacuum}}<br /> ** {{messageLink|1649062270.289865713@f403.i.mail.ru|Thread with patch}} [https://commitfest.postgresql.org/43/3739/ CF Entry]<br /> <br /> * {{messageLink|1516594.1681482708@sss.pgh.pa.us|We are not compatible with newly-released LLVM 16}}<br /> ** {{messageLink|CA%2BhUKGKNX_%3Df%2B1C4r06WETKTq0G4Z_7q4L4Fxn5WWpMycDj9Fw%40mail.gmail.com|Patch}}<br /> ** Owner: Thomas Munro (volunteer LLVM API change chaser)<br /> <br /> * {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br /> ** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|&quot;PANIC: could not open critical system index 2662&quot; - twice}}<br /> <br /> === Fixed issues ===<br /> <br /> * [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and &gt;MaxAllocSize lengths]<br /> ** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br /> ** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br /> ** Affects all stable branches.<br /> ** Fixed at: {{PgCommitURL|8fcb32db98eda1ad2a0c0b40b1cbb5d9a7aa68f0}} and {{PgCommitURL|ffd1b6bb6f8a2ffc929699772610c6925364dbb3}} for HEAD.<br /> <br /> * [https://www.postgresql.org/message-id/flat/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Fix fseek() detection of unseekable files on WIN32]<br /> ** Fixed at: {{PgCommitURL|a923e21631a29dc8b8781d7d02b5003d0df64ca3}} and {{PgCommitURL|765f5df726918bcdcfd16bcc5418e48663d1dd59}}, down to 14.<br /> <br /> * {{messageLink|CAAKRu_bETD%2BAri600h6fRjX2p8rJSeMAUp%3D_y88juqOZgouTSg%40mail.gmail.com|Can't disable autovacuum cost delay through storage parameter}}<br /> ** Fixed at: {{PgCommitURL|bfac8f8bc4a44c67c9f35b5266676278e4ba1217}}, down to 11.<br /> <br /> * {{messageLink|CAJ7c6TMBTN3rcz4%3DAjYhLPD_w3FFT0Wq_C15jxCDn8U4tZnH1g@mail.gmail.com| EPQ misbehaves for inherited/partitioned tables}}<br /> ** Fixed at: {{PgCommitURL|70b42f279}}, down to 14.<br /> <br /> == Non-bugs ==<br /> <br /> * {{messageLink|17862-1ab8f74b0f7b0611@postgresql.org|WindowAgg startup costs don't take into account partition bound. Can lead to incorrect use of cheap startup plans}}<br /> ** {{messageLink|CAApHDvrB0S5BMv+0-wTTqWFE-BJ0noWqTnDu9QQfjZ2VSpLv_g@mail.gmail.com|Patch to fix and discussion}}<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 16beta2 ===<br /> * {{messageLink|CAMbWs4_tuVn9EwwMcggGiZJWWstdXX_ci8FeEU17vs+4nLgw3w@mail.gmail.com|Assert failure and wrong query results due to incorrectly removing PHV}}<br /> ** Owner: Tom Lane<br /> ** Fixed at: {{PgCommitURL|9a2dbc614e6e47da3c49daacec106da32eba9467}}<br /> * {{messageLink|CAMbWs4-_vwkBij4XOQ5ukxUvLgwTm0kS5_DO9CicUeKbEfKjUw%40mail.gmail.com|Assert failure of the cross-check for nullingrels}}<br /> ** Owner: Tom Lane<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** [https://commitfest.postgresql.org/43/4250/ CF Entry]<br /> ** Fixed at: {{PgCommitURL|991a3df22}}<br /> <br /> === resolved before 16beta1 ===<br /> * {{messageLink|CAHewXNnu7u1aT%3D%3DWjnCRa%2BSzKb6s80hvwPP_9eMvvvtdyFdqjw%40mail.gmail.com|ERROR: wrong varnullingrels (b 5 7) (expected (b)) for Var 3/3}}<br /> ** Fixed at: {{PgCommitURL|d0f952691}}<br /> * {{messageLink|d46f9265-ff3c-6743-2278-6772598233c2%40pgmasters.net|Possible regression setting GUCs on \connect}}<br /> ** Owner: Alexander Korotkov<br /> ** Discussion on reverting {{PgCommitURL|096dd80f3}}<br /> ** Original commit: {{PgCommitURL|096dd80f3}}<br /> ** Reverted at: {{PgCommitURL|b9a7a822723aebb16cbe7e5fb874e5124745b07e}}<br /> <br /> * Planner makes improper clause pushdown decisions due to outer-join-aware-Vars changes<br /> ** {{messageLink|0b819232-4b50-f245-1c7d-c8c61bf41827@postgrespro.ru|Clause accidentally pushed down}}<br /> ** {{messageLink|CAHewXNks3w_Vy9CWoVtHx1XSaeiFpsOzh-zy5eu0Khp1PtG1sA@mail.gmail.com|wrong results due to qual pushdown}}<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** Fixed at: {{PgCommitURL|9df8f903eb6758be5a19e66cdf77e922e9329c31}}<br /> <br /> * Revert {{PgCommitURL|ec386948948}}, per {{messageLink|20230330105325.y6uvpalspynf2frt@alvherre.pgsql|Re: &quot;variable not found in subplan target list&quot;}}<br /> ** Reverted at {{PgCommitURL|5472743d9e8}}<br /> <br /> * [https://www.postgresql.org/message-id/CAEZATCWETioXs5kY8vT6BVguY41_wD962VDk%3Du_Nvd7S1UXzuQ%40mail.gmail.com ERROR: ORDER/GROUP BY expression not found in targetlist]<br /> ** Fixed at: {{PgCommitURL|da5800d5fa636c6e10c9c98402d872c76aa1c8d0}}<br /> <br /> * [https://www.postgresql.org/message-id/20230212233711.GA1316@telsasoft.com various elogs hit by sqlsmith (ExecRTCheckPerms() and many prunable partitions)]<br /> ** Fixed at: {{PgCommitURL|c7468c73f7b6e842a53c12eaee5578a76a8fa7a6}}<br /> <br /> * [https://www.postgresql.org/message-id/20230228235834.GC30529@telsasoft.com pg_dump: zlib compression fails for empty objects (LOs)]<br /> ** Fixed at: {{PgCommitURL|00d9dcf5bebbb355152a60f0e2120cdf7f9e7ddd}}<br /> <br /> * [https://www.postgresql.org/message-id/20230227044910.GO1653@telsasoft.com pg_dump: lz4 compression uses no persistent state and writes a block header for every row]<br /> ** Fixed at: {{PgCommitURL|0070b66fef21e909adb283f7faa7b1978836ad75}}<br /> <br /> * {{messageLink|3590249.1680971629@sss.pgh.pa.us|Assertion failure with parallel full hash join}}<br /> ** Fixed at: {{PgCommitURL|b37d051b0e59e4324e346655a27509507813db79}}<br /> <br /> * {{messageLink|ZDDO6jaESKaBgej0@tamriel.snowman.net|De-revert &quot;Add support for Kerberos credential delegation&quot;}}<br /> ** Owner: Stephen Frost<br /> ** Original commit: {{PgCommitURL|3d4fa227bce4294ce1cc214b4a9d3b7caa3f0454}}<br /> ** Revert: ({{PgCommitURL|3d03b24c350ab060bb223623bdff38835bd7afd0}}<br /> ** De-Revert: {{PgCommitURL|6633cfb21691840c33816a6dacaca0b504efb895}}<br /> ** Resolved at: {{PgCommitURL|f7431bca8b0138bdbce7025871560d39119565a0}}<br /> <br /> * {{messageLink|c39be3c5-c1a5-1e33-1024-16f527e251a4@enterprisedb.com|SSL tests break on non-existing system CA pool}}<br /> ** Fixed at: {{PgCommitURL|0b5d1fb36adda612bd3d5d032463a6eeb0729237}}<br /> <br /> * {{messageLink|CAD21AoBS7o6Ljt_vfqPQPf67AhzKu3fR0iqk8B%3DvVYczMugKMQ%40mail.gmail.com|VacuumUpdateCosts() logging condition incorrect for some initial values of vacuum_cost_delay}}<br /> ** Fixed at: {{PgCommitURL|a9781ae11ba2fdb44a3a72c9a7ebb727140b25c5}}<br /> <br /> * {{messageLink|CA%2BhUKGJ-ZPJwKHVLbqye92-ZXeLoCHu5wJL6L6HhNP7FkJ%3DmeA%40mail.gmail.com|check_strxfrm_bug()}}<br /> ** Owner: Thomas Munro<br /> ** Fixed at: {{PgCommitURL|7d3d72b55edd1b7552a9a358991555994efab0e9}}<br /> <br /> * {{messageLink|20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de|Should we remove vacuum_defer_cleanup_age?}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|1118cd37eb61e6a2428f457a8b2026a7bb3f801a}}<br /> <br /> * {{messageLink|2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com|Add two missing tests in 035_standby_logical_decoding.pl}}<br /> ** Fixed at: {{PgCommitURL|376dc820531bafcbf105fff74c5b14c23d9950af}}<br /> ** Fixed at: {{PgCommitURL|a6e04b1d20c2e9cece9b64bb5b36ebfdc3a9031b}}<br /> <br /> * {{messageLink|b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com|Performance regression due to SQLValueFunction removal}}<br /> ** Fixed at: {{PgCommitURL|d8c3106bb60e4f87be595f241e173ba3c2b7aa2c}}<br /> <br /> * {{messageLink|20230419172326.dhgyo4wrrhulovt6%40awork3.anarazel.de|pg_stat_io not tracking smgrwriteback() is confusing}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|093e5c57d506783a95dd8feddd9a3f2651e1aeba}}<br /> <br /> * {{messageLink|ZFhCyn4Gm2eu60rB@paquier.xyz|Table data compression is broken with pg_dump --compress lz4}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|1a05c1d252993b0a59c58a6daf91a2df9333044f}}<br /> <br /> * {{messageLink|94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com|Valgrind unhappy with LZ4F code in pg_dump}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|3c18d90f8907e53c3021fca13ad046133c480e4d}}<br /> <br /> * {{messageLink|20230509190247.3rrplhdgem6su6cg@awork3.anarazel.de|walsender performance regression due to logical decoding on standby changes}}<br /> ** Owner: Andres Freund<br /> ** Original commit: {{PgCommitURL|e101dfac}}<br /> ** Fixed at: {{PgCommitURL|bc971f4025c378ce500d86597c34b0ef996d4d8c}}<br /> <br /> == Won't Fix ==<br /> <br /> * Is it OK that WL_SOCKET_ACCEPT is less fair on Windows than on Unix (and than the coding before 16) when there are multiple server sockets configured?<br /> ** {{messageLink|CA%2BhUKG%2BA2dk29hr5zRP3HVJQ-_PncNJM6HVQ7aaYLXLRBZU-xw%40mail.gmail.com|WL_SOCKET_ACCEPT fairness on Windows}} has a (blind) patch to fix that, but would need a Windows hacker to test<br /> ** Owner: Thomas Munro<br /> ** Original commit: {{PgCommitURL|7389aad6}}<br /> ** Issue reclassified as a non-critical improvement to be [https://commitfest.postgresql.org/43/4263/ considered for 17]<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Beta 2: TBD<br /> * Beta 1: May 25, 2023<br /> * Feature Freeze: April 8, 2023 0:00 AoE ('''Last Day to Commit Features''')<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 15 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_16_Open_Items&diff=37847 2023-05-21T18:04:17Z

<p>Andresfreund: Mark walsender issue as fixed</p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Switch to ICU for 17?<br /> ** Owner: Jeff Davis<br /> ** {{messageLink|82c4c816-06f6-d3e3-ba02-fca4a5cef065@enterprisedb.com|I suggest waiting until next week to commit it and then see what happens}}<br /> ** [https://commitfest.postgresql.org/42/4169/ CF Entry]<br /> * {{messageLink|e587e2ee-7de0-88a2-10f8-c7cf001bab8c%40postgrespro.ru|psql: Add role's membership options to the \du+ command}}<br /> ** [https://commitfest.postgresql.org/43/4116/ CF Entry]<br /> ** NOTE: This is not a committed feature for v16<br /> * {{messageLink|874jp9f5jo.fsf@news-spur.riddles.org.uk|The rules for choosing default ICU locale seem pretty unfriendly}}<br /> ** Owner: Jeff Davis<br /> * {{messageLink|CAMbWs4-_vwkBij4XOQ5ukxUvLgwTm0kS5_DO9CicUeKbEfKjUw%40mail.gmail.com|Assert failure of the cross-check for nullingrels}}<br /> ** Owner: Tom Lane<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** [https://commitfest.postgresql.org/43/4250/ CF Entry]<br /> * {{messageLink|CAHewXNnu7u1aT%3D%3DWjnCRa%2BSzKb6s80hvwPP_9eMvvvtdyFdqjw%40mail.gmail.com|ERROR: wrong varnullingrels (b 5 7) (expected (b)) for Var 3/3}}<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * [https://www.postgresql.org/message-id/268fd337-8bb7-92e6-0da2-416c022c11f3%40enterprisedb.com Reconsider a utility_query_id GUC to control if query jumbling of utilities can go through the past string-only mode and the new mode?]<br /> ** Potential owner: Michael Paquier<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br /> ** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br /> ** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br /> ** Unclear what the user impact is.<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br /> ** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer] [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report] [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report] [https://www.postgresql.org/message-id/flat/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA%40mail.gmail.com Another report] [https://www.postgresql.org/message-id/flat/a53cacb0-8835-57d6-31e4-4c5ef196de1a@deepbluecap.com Another report]<br /> <br /> * [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br /> ** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/ZArVOMifjzE7f8W7%40paquier.xyz Requiring recovery.signal or standby.signal when recovering with a backup_label]<br /> ** This is a rather old behavior that affects all stable branches, still not something that should be backpatched as-is.<br /> <br /> * {{messageLink|cfcca574-6967-c5ab-7dc3-2c82b6723b99@mail.ru|pg_visibility's pg_check_visible() yields false positive when working in parallel with autovacuum}}<br /> ** {{messageLink|1649062270.289865713@f403.i.mail.ru|Thread with patch}} [https://commitfest.postgresql.org/43/3739/ CF Entry]<br /> <br /> * {{messageLink|17862-1ab8f74b0f7b0611@postgresql.org|WindowAgg startup costs don't take into account partition bound. Can lead to incorrect use of cheap startup plans}}<br /> ** {{messageLink|CAApHDvrB0S5BMv+0-wTTqWFE-BJ0noWqTnDu9QQfjZ2VSpLv_g@mail.gmail.com|Patch to fix and discussion}}<br /> <br /> * {{messageLink|1516594.1681482708@sss.pgh.pa.us|We are not compatible with newly-released LLVM 16}}<br /> <br /> * {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br /> ** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|&quot;PANIC: could not open critical system index 2662&quot; - twice}}<br /> <br /> * {{messageLink|CAJ7c6TMBTN3rcz4%3DAjYhLPD_w3FFT0Wq_C15jxCDn8U4tZnH1g@mail.gmail.com| EPQ misbehaves for inherited/partitioned tables}}<br /> ** Owner: Tom Lane (86dc90056)<br /> <br /> === Fixed issues ===<br /> <br /> * [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and &gt;MaxAllocSize lengths]<br /> ** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br /> ** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br /> ** Affects all stable branches.<br /> ** Fixed at: {{PgCommitURL|8fcb32db98eda1ad2a0c0b40b1cbb5d9a7aa68f0}} and {{PgCommitURL|ffd1b6bb6f8a2ffc929699772610c6925364dbb3}} for HEAD.<br /> <br /> * [https://www.postgresql.org/message-id/flat/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Fix fseek() detection of unseekable files on WIN32]<br /> ** Fixed at: {{PgCommitURL|a923e21631a29dc8b8781d7d02b5003d0df64ca3}} and {{PgCommitURL|765f5df726918bcdcfd16bcc5418e48663d1dd59}}, down to 14.<br /> <br /> * {{messageLink|CAAKRu_bETD%2BAri600h6fRjX2p8rJSeMAUp%3D_y88juqOZgouTSg%40mail.gmail.com|Can't disable autovacuum cost delay through storage parameter}}<br /> ** Fixed at: {{PgCommitURL|bfac8f8bc4a44c67c9f35b5266676278e4ba1217}}, down to 11.<br /> <br /> == Non-bugs ==<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 16beta1 ===<br /> <br /> * {{messageLink|d46f9265-ff3c-6743-2278-6772598233c2%40pgmasters.net|Possible regression setting GUCs on \connect}}<br /> ** Owner: Alexander Korotkov<br /> ** Discussion on reverting {{PgCommitURL|096dd80f3}}<br /> ** Original commit: {{PgCommitURL|096dd80f3}}<br /> ** Fixed at (bug only): {{PgCommitURL|db93e739ac61332126207b16f14da93f8ecac594}}<br /> ** Fixed at (feature reverted): {{PgCommitURL|b9a7a822723aebb16cbe7e5fb874e5124745b07e}}<br /> <br /> * Planner makes improper clause pushdown decisions due to outer-join-aware-Vars changes<br /> ** {{messageLink|0b819232-4b50-f245-1c7d-c8c61bf41827@postgrespro.ru|Clause accidentally pushed down}}<br /> ** {{messageLink|CAHewXNks3w_Vy9CWoVtHx1XSaeiFpsOzh-zy5eu0Khp1PtG1sA@mail.gmail.com|wrong results due to qual pushdown}}<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** Fixed at: {{PgCommitURL|9df8f903eb6758be5a19e66cdf77e922e9329c31}}<br /> <br /> * Revert {{PgCommitURL|ec386948948}}, per {{messageLink|20230330105325.y6uvpalspynf2frt@alvherre.pgsql|Re: &quot;variable not found in subplan target list&quot;}}<br /> ** Reverted at {{PgCommitURL|5472743d9e8}}<br /> <br /> * [https://www.postgresql.org/message-id/CAEZATCWETioXs5kY8vT6BVguY41_wD962VDk%3Du_Nvd7S1UXzuQ%40mail.gmail.com ERROR: ORDER/GROUP BY expression not found in targetlist]<br /> ** Fixed at: {{PgCommitURL|da5800d5fa636c6e10c9c98402d872c76aa1c8d0}}<br /> <br /> * [https://www.postgresql.org/message-id/20230212233711.GA1316@telsasoft.com various elogs hit by sqlsmith (ExecRTCheckPerms() and many prunable partitions)]<br /> ** Fixed at: {{PgCommitURL|c7468c73f7b6e842a53c12eaee5578a76a8fa7a6}}<br /> <br /> * [https://www.postgresql.org/message-id/20230228235834.GC30529@telsasoft.com pg_dump: zlib compression fails for empty objects (LOs)]<br /> ** Fixed at: {{PgCommitURL|00d9dcf5bebbb355152a60f0e2120cdf7f9e7ddd}}<br /> <br /> * [https://www.postgresql.org/message-id/20230227044910.GO1653@telsasoft.com pg_dump: lz4 compression uses no persistent state and writes a block header for every row]<br /> ** Fixed at: {{PgCommitURL|0070b66fef21e909adb283f7faa7b1978836ad75}}<br /> <br /> * {{messageLink|3590249.1680971629@sss.pgh.pa.us|Assertion failure with parallel full hash join}}<br /> ** Fixed at: {{PgCommitURL|b37d051b0e59e4324e346655a27509507813db79}}<br /> <br /> * {{messageLink|ZDDO6jaESKaBgej0@tamriel.snowman.net|De-revert &quot;Add support for Kerberos credential delegation&quot;}}<br /> ** Owner: Stephen Frost<br /> ** Original commit: {{PgCommitURL|3d4fa227bce4294ce1cc214b4a9d3b7caa3f0454}}<br /> ** Revert: ({{PgCommitURL|3d03b24c350ab060bb223623bdff38835bd7afd0}}<br /> ** De-Revert: {{PgCommitURL|6633cfb21691840c33816a6dacaca0b504efb895}}<br /> ** Resolved at: {{PgCommitURL|f7431bca8b0138bdbce7025871560d39119565a0}}<br /> <br /> * {{messageLink|c39be3c5-c1a5-1e33-1024-16f527e251a4@enterprisedb.com|SSL tests break on non-existing system CA pool}}<br /> ** Fixed at: {{PgCommitURL|0b5d1fb36adda612bd3d5d032463a6eeb0729237}}<br /> <br /> * {{messageLink|CAD21AoBS7o6Ljt_vfqPQPf67AhzKu3fR0iqk8B%3DvVYczMugKMQ%40mail.gmail.com|VacuumUpdateCosts() logging condition incorrect for some initial values of vacuum_cost_delay}}<br /> ** Fixed at: {{PgCommitURL|a9781ae11ba2fdb44a3a72c9a7ebb727140b25c5}}<br /> <br /> * {{messageLink|CA%2BhUKGJ-ZPJwKHVLbqye92-ZXeLoCHu5wJL6L6HhNP7FkJ%3DmeA%40mail.gmail.com|check_strxfrm_bug()}}<br /> ** Owner: Thomas Munro<br /> ** Fixed at: {{PgCommitURL|7d3d72b55edd1b7552a9a358991555994efab0e9}}<br /> <br /> * {{messageLink|20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de|Should we remove vacuum_defer_cleanup_age?}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|1118cd37eb61e6a2428f457a8b2026a7bb3f801a}}<br /> <br /> * {{messageLink|2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com|Add two missing tests in 035_standby_logical_decoding.pl}}<br /> ** Fixed at: {{PgCommitURL|376dc820531bafcbf105fff74c5b14c23d9950af}}<br /> ** Fixed at: {{PgCommitURL|a6e04b1d20c2e9cece9b64bb5b36ebfdc3a9031b}}<br /> <br /> * {{messageLink|b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com|Performance regression due to SQLValueFunction removal}}<br /> ** Fixed at: {{PgCommitURL|d8c3106bb60e4f87be595f241e173ba3c2b7aa2c}}<br /> <br /> * {{messageLink|20230419172326.dhgyo4wrrhulovt6%40awork3.anarazel.de|pg_stat_io not tracking smgrwriteback() is confusing}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|093e5c57d506783a95dd8feddd9a3f2651e1aeba}}<br /> <br /> * {{messageLink|ZFhCyn4Gm2eu60rB@paquier.xyz|Table data compression is broken with pg_dump --compress lz4}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|1a05c1d252993b0a59c58a6daf91a2df9333044f}}<br /> <br /> * {{messageLink|94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com|Valgrind unhappy with LZ4F code in pg_dump}}<br /> ** Owner: Tomas Vondra<br /> ** Fixed at: {{PgCommitURL|3c18d90f8907e53c3021fca13ad046133c480e4d}}<br /> <br /> * {{messageLink|20230509190247.3rrplhdgem6su6cg@awork3.anarazel.de|walsender performance regression due to logical decoding on standby changes}}<br /> ** Owner: Andres Freund<br /> ** Original commit: {{PgCommitURL|e101dfac}}<br /> ** Fixed at: {{PgCommitURL|bc971f4025c378ce500d86597c34b0ef996d4d8c}}<br /> <br /> <br /> == Won't Fix ==<br /> <br /> * Is it OK that WL_SOCKET_ACCEPT is less fair on Windows than on Unix (and than the coding before 16) when there are multiple server sockets configured?<br /> ** {{messageLink|CA%2BhUKG%2BA2dk29hr5zRP3HVJQ-_PncNJM6HVQ7aaYLXLRBZU-xw%40mail.gmail.com|WL_SOCKET_ACCEPT fairness on Windows}} has a (blind) patch to fix that, but would need a Windows hacker to test<br /> ** Owner: Thomas Munro<br /> ** Original commit: {{PgCommitURL|7389aad6}}<br /> ** Issue reclassified as a non-critical improvement to be [https://commitfest.postgresql.org/43/4263/ considered for 17]<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Beta 2: TBD<br /> * Beta 1: May 25, 2023<br /> * Feature Freeze: April 8, 2023 0:00 AoE ('''Last Day to Commit Features''')<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 15 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_16_Open_Items&diff=37833 2023-05-17T19:29:02Z

<p>Andresfreund: </p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Switch to ICU for 17?<br /> ** Owner: Jeff Davis<br /> ** {{messageLink|82c4c816-06f6-d3e3-ba02-fca4a5cef065@enterprisedb.com|I suggest waiting until next week to commit it and then see what happens}}<br /> ** [https://commitfest.postgresql.org/42/4169/ CF Entry]<br /> * {{messageLink|e587e2ee-7de0-88a2-10f8-c7cf001bab8c%40postgrespro.ru|psql: Add role's membership options to the \du+ command}}<br /> ** [https://commitfest.postgresql.org/43/4116/ CF Entry]<br /> ** NOTE: This is not a committed feature for v16<br /> * {{messageLink|874jp9f5jo.fsf@news-spur.riddles.org.uk|The rules for choosing default ICU locale seem pretty unfriendly}}<br /> ** Owner: Jeff Davis<br /> * {{messageLink|94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com|Valgrind unhappy with LZ4F code in pg_dump}}<br /> ** Owner: Tomas Vondra<br /> * {{messageLink|ZFhCyn4Gm2eu60rB@paquier.xyz|Table data compression is broken with pg_dump --compress lz4}}<br /> ** Owner: Tomas Vondra<br /> * {{messageLink|CAMbWs4-_vwkBij4XOQ5ukxUvLgwTm0kS5_DO9CicUeKbEfKjUw%40mail.gmail.com|Assert failure of the cross-check for nullingrels}}<br /> ** Owner: Tom Lane<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** [https://commitfest.postgresql.org/43/4250/ CF Entry]<br /> * {{messageLink|20230509190247.3rrplhdgem6su6cg@awork3.anarazel.de|walsender performance regression due to logical decoding on standby changes}}<br /> ** Owner: Andres Freund<br /> ** Original commit: {{PgCommitURL|e101dfac}}<br /> ** {{messageLink|CALj2ACWeo64RSqf8tDbnSSUm_vbpK5GYdDiiFQk8E3Fg38mBdw@mail.gmail.com|Patch sent}}<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * [https://www.postgresql.org/message-id/268fd337-8bb7-92e6-0da2-416c022c11f3%40enterprisedb.com Reconsider a utility_query_id GUC to control if query jumbling of utilities can go through the past string-only mode and the new mode?]<br /> ** Potential owner: Michael Paquier<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br /> ** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br /> ** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br /> ** Unclear what the user impact is.<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br /> ** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer] [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report] [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report] [https://www.postgresql.org/message-id/flat/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA%40mail.gmail.com Another report] [https://www.postgresql.org/message-id/flat/a53cacb0-8835-57d6-31e4-4c5ef196de1a@deepbluecap.com Another report]<br /> <br /> * [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br /> ** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/ZArVOMifjzE7f8W7%40paquier.xyz Requiring recovery.signal or standby.signal when recovering with a backup_label]<br /> ** This is a rather old behavior that affects all stable branches, still not something that should be backpatched as-is.<br /> <br /> * {{messageLink|cfcca574-6967-c5ab-7dc3-2c82b6723b99@mail.ru|pg_visibility's pg_check_visible() yields false positive when working in parallel with autovacuum}}<br /> ** {{messageLink|1649062270.289865713@f403.i.mail.ru|Thread with patch}} [https://commitfest.postgresql.org/43/3739/ CF Entry]<br /> <br /> * {{messageLink|17862-1ab8f74b0f7b0611@postgresql.org|WindowAgg startup costs don't take into account partition bound. Can lead to incorrect use of cheap startup plans}}<br /> ** {{messageLink|CAApHDvrB0S5BMv+0-wTTqWFE-BJ0noWqTnDu9QQfjZ2VSpLv_g@mail.gmail.com|Patch to fix and discussion}}<br /> <br /> * {{messageLink|1516594.1681482708@sss.pgh.pa.us|We are not compatible with newly-released LLVM 16}}<br /> <br /> * {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br /> ** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|&quot;PANIC: could not open critical system index 2662&quot; - twice}}<br /> <br /> === Fixed issues ===<br /> <br /> * [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and &gt;MaxAllocSize lengths]<br /> ** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br /> ** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br /> ** Affects all stable branches.<br /> ** Fixed at: {{PgCommitURL|8fcb32db98eda1ad2a0c0b40b1cbb5d9a7aa68f0}} and {{PgCommitURL|ffd1b6bb6f8a2ffc929699772610c6925364dbb3}} for HEAD.<br /> <br /> * [https://www.postgresql.org/message-id/flat/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Fix fseek() detection of unseekable files on WIN32]<br /> ** Fixed at: {{PgCommitURL|a923e21631a29dc8b8781d7d02b5003d0df64ca3}} and {{PgCommitURL|765f5df726918bcdcfd16bcc5418e48663d1dd59}}, down to 14.<br /> <br /> * {{messageLink|CAAKRu_bETD%2BAri600h6fRjX2p8rJSeMAUp%3D_y88juqOZgouTSg%40mail.gmail.com|Can't disable autovacuum cost delay through storage parameter}}<br /> ** Fixed at: {{PgCommitURL|bfac8f8bc4a44c67c9f35b5266676278e4ba1217}}, down to 11.<br /> <br /> == Non-bugs ==<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 16beta1 ===<br /> <br /> * {{messageLink|d46f9265-ff3c-6743-2278-6772598233c2%40pgmasters.net|Possible regression setting GUCs on \connect}}<br /> ** Owner: Alexander Korotkov<br /> ** Discussion on reverting {{PgCommitURL|096dd80f3}}<br /> ** Original commit: {{PgCommitURL|096dd80f3}}<br /> ** Fixed at (bug only): {{PgCommitURL|db93e739ac61332126207b16f14da93f8ecac594}}<br /> ** Fixed at (feature reverted): {{PgCommitURL|b9a7a822723aebb16cbe7e5fb874e5124745b07e}}<br /> <br /> * Planner makes improper clause pushdown decisions due to outer-join-aware-Vars changes<br /> ** {{messageLink|0b819232-4b50-f245-1c7d-c8c61bf41827@postgrespro.ru|Clause accidentally pushed down}}<br /> ** {{messageLink|CAHewXNks3w_Vy9CWoVtHx1XSaeiFpsOzh-zy5eu0Khp1PtG1sA@mail.gmail.com|wrong results due to qual pushdown}}<br /> ** Original commit: {{PgCommitURL|2489d76c4}}<br /> ** Fixed at: {{PgCommitURL|9df8f903eb6758be5a19e66cdf77e922e9329c31}}<br /> <br /> * Revert {{PgCommitURL|ec386948948}}, per {{messageLink|20230330105325.y6uvpalspynf2frt@alvherre.pgsql|Re: &quot;variable not found in subplan target list&quot;}}<br /> ** Reverted at {{PgCommitURL|5472743d9e8}}<br /> <br /> * [https://www.postgresql.org/message-id/CAEZATCWETioXs5kY8vT6BVguY41_wD962VDk%3Du_Nvd7S1UXzuQ%40mail.gmail.com ERROR: ORDER/GROUP BY expression not found in targetlist]<br /> ** Fixed at: {{PgCommitURL|da5800d5fa636c6e10c9c98402d872c76aa1c8d0}}<br /> <br /> * [https://www.postgresql.org/message-id/20230212233711.GA1316@telsasoft.com various elogs hit by sqlsmith (ExecRTCheckPerms() and many prunable partitions)]<br /> ** Fixed at: {{PgCommitURL|c7468c73f7b6e842a53c12eaee5578a76a8fa7a6}}<br /> <br /> * [https://www.postgresql.org/message-id/20230228235834.GC30529@telsasoft.com pg_dump: zlib compression fails for empty objects (LOs)]<br /> ** Fixed at: {{PgCommitURL|00d9dcf5bebbb355152a60f0e2120cdf7f9e7ddd}}<br /> <br /> * [https://www.postgresql.org/message-id/20230227044910.GO1653@telsasoft.com pg_dump: lz4 compression uses no persistent state and writes a block header for every row]<br /> ** Fixed at: {{PgCommitURL|0070b66fef21e909adb283f7faa7b1978836ad75}}<br /> <br /> * {{messageLink|3590249.1680971629@sss.pgh.pa.us|Assertion failure with parallel full hash join}}<br /> ** Fixed at: {{PgCommitURL|b37d051b0e59e4324e346655a27509507813db79}}<br /> <br /> * {{messageLink|ZDDO6jaESKaBgej0@tamriel.snowman.net|De-revert &quot;Add support for Kerberos credential delegation&quot;}}<br /> ** Owner: Stephen Frost<br /> ** Original commit: {{PgCommitURL|3d4fa227bce4294ce1cc214b4a9d3b7caa3f0454}}<br /> ** Revert: ({{PgCommitURL|3d03b24c350ab060bb223623bdff38835bd7afd0}}<br /> ** De-Revert: {{PgCommitURL|6633cfb21691840c33816a6dacaca0b504efb895}}<br /> ** Resolved at: {{PgCommitURL|f7431bca8b0138bdbce7025871560d39119565a0}}<br /> <br /> * {{messageLink|c39be3c5-c1a5-1e33-1024-16f527e251a4@enterprisedb.com|SSL tests break on non-existing system CA pool}}<br /> ** Fixed at: {{PgCommitURL|0b5d1fb36adda612bd3d5d032463a6eeb0729237}}<br /> <br /> * {{messageLink|CAD21AoBS7o6Ljt_vfqPQPf67AhzKu3fR0iqk8B%3DvVYczMugKMQ%40mail.gmail.com|VacuumUpdateCosts() logging condition incorrect for some initial values of vacuum_cost_delay}}<br /> ** Fixed at: {{PgCommitURL|a9781ae11ba2fdb44a3a72c9a7ebb727140b25c5}}<br /> <br /> * {{messageLink|CA%2BhUKGJ-ZPJwKHVLbqye92-ZXeLoCHu5wJL6L6HhNP7FkJ%3DmeA%40mail.gmail.com|check_strxfrm_bug()}}<br /> ** Owner: Thomas Munro<br /> ** Fixed at: {{PgCommitURL|7d3d72b55edd1b7552a9a358991555994efab0e9}}<br /> <br /> * {{messageLink|20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de|Should we remove vacuum_defer_cleanup_age?}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|1118cd37eb61e6a2428f457a8b2026a7bb3f801a}}<br /> <br /> * {{messageLink|2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com|Add two missing tests in 035_standby_logical_decoding.pl}}<br /> ** Fixed at: {{PgCommitURL|376dc820531bafcbf105fff74c5b14c23d9950af}}<br /> ** Fixed at: {{PgCommitURL|a6e04b1d20c2e9cece9b64bb5b36ebfdc3a9031b}}<br /> <br /> * {{messageLink|b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com|Performance regression due to SQLValueFunction removal}}<br /> ** Fixed at: {{PgCommitURL|d8c3106bb60e4f87be595f241e173ba3c2b7aa2c}}<br /> <br /> * {{messageLink|20230419172326.dhgyo4wrrhulovt6%40awork3.anarazel.de|pg_stat_io not tracking smgrwriteback() is confusing}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|093e5c57d506783a95dd8feddd9a3f2651e1aeba}}<br /> <br /> == Won't Fix ==<br /> <br /> * Is it OK that WL_SOCKET_ACCEPT is less fair on Windows than on Unix (and than the coding before 16) when there are multiple server sockets configured?<br /> ** {{messageLink|CA%2BhUKG%2BA2dk29hr5zRP3HVJQ-_PncNJM6HVQ7aaYLXLRBZU-xw%40mail.gmail.com|WL_SOCKET_ACCEPT fairness on Windows}} has a (blind) patch to fix that, but would need a Windows hacker to test<br /> ** Owner: Thomas Munro<br /> ** Original commit: {{PgCommitURL|7389aad6}}<br /> ** Issue reclassified as a non-critical improvement to be [https://commitfest.postgresql.org/43/4263/ considered for 17]<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Beta 2: TBD<br /> * Beta 1: May 25, 2023<br /> * Feature Freeze: April 8, 2023 0:00 AoE ('''Last Day to Commit Features''')<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 15 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PostgreSQL_16_Open_Items&diff=37815 2023-05-09T16:48:02Z

<p>Andresfreund: </p> <hr /> <div>== Open Issues ==<br /> <br /> '''NOTE''': Please place new open items at the end of the list.<br /> <br /> '''NOTE''': If known, please list the Owner of the open item.<br /> <br /> * Is it OK that WL_SOCKET_ACCEPT is less fair on Windows than on Unix (and than the coding before 16) when there are multiple server sockets configured?<br /> ** {{messageLink|CA%2BhUKG%2BA2dk29hr5zRP3HVJQ-_PncNJM6HVQ7aaYLXLRBZU-xw%40mail.gmail.com|WL_SOCKET_ACCEPT fairness on Windows}} has a (blind) patch to fix that, but would need a Windows hacker to test<br /> * Planner makes improper clause pushdown decisions due to outer-join-aware-Vars changes<br /> ** {{messageLink|0b819232-4b50-f245-1c7d-c8c61bf41827@postgrespro.ru|Clause accidentally pushed down}}<br /> ** {{messageLink|CAHewXNks3w_Vy9CWoVtHx1XSaeiFpsOzh-zy5eu0Khp1PtG1sA@mail.gmail.com|wrong results due to qual pushdown}}<br /> ** Owner: Tom Lane (2489d76c4)<br /> * Switch to ICU for 17?<br /> ** {{messageLink|82c4c816-06f6-d3e3-ba02-fca4a5cef065@enterprisedb.com|I suggest waiting until next week to commit it and then see what happens}}<br /> ** [https://commitfest.postgresql.org/42/4169/ CF Entry]<br /> * {{messageLink|e587e2ee-7de0-88a2-10f8-c7cf001bab8c%40postgrespro.ru|psql: Add role's membership options to the \du+ command}}<br /> ** [https://commitfest.postgresql.org/43/4116/ CF Entry]<br /> ** NOTE: This is not a committed feature for v16<br /> * {{messageLink|874jp9f5jo.fsf@news-spur.riddles.org.uk|The rules for choosing default ICU locale seem pretty unfriendly}}<br /> * {{messageLink|20230419172326.dhgyo4wrrhulovt6%40awork3.anarazel.de|pg_stat_io not tracking smgrwriteback() is confusing}}<br /> * {{messageLink|d46f9265-ff3c-6743-2278-6772598233c2%40pgmasters.net|Possible regression setting GUCs on \connect}}<br /> ** Owner: Alexander Korotkov<br /> ** Discussion on reverting {{PgCommitURL|096dd80f3}}<br /> ** Original commit: {{PgCommitURL|096dd80f3}}<br /> ** Fixed at (bug only): {{PgCommitURL|db93e739ac61332126207b16f14da93f8ecac594}}<br /> * {{messageLink|94ae9bca-5ebb-1e68-bb7b-4f32e89fefbe@gmail.com|Valgrind unhappy with LZ4F code in pg_dump}}<br /> ** Owner: Tomas Vondra<br /> * {{messageLink|ZFhCyn4Gm2eu60rB@paquier.xyz|Table data compression is broken with pg_dump --compress lz4}}<br /> ** Owner: Tomas Vondra<br /> * {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br /> ** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|&quot;PANIC: could not open critical system index 2662&quot; - twice}}<br /> ** v16-specific fix<br /> * {{messageLink|b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com|Performance regression due to SQLValueFunction removal}}<br /> ** Owner: Michael Paquier<br /> ** Original commit: {{PgCommitURL|f193883fc9c}}<br /> <br /> == Decisions to Recheck Mid-Beta ==<br /> <br /> * [https://www.postgresql.org/message-id/268fd337-8bb7-92e6-0da2-416c022c11f3%40enterprisedb.com Reconsider a utility_query_id GUC to control if query jumbling of utilities can go through the past string-only mode and the new mode?]<br /> ** Potential owner: Michael Paquier<br /> <br /> == Older bugs affecting stable branches ==<br /> <br /> === Live issues ===<br /> <br /> * [https://www.postgresql.org/message-id/flat/CA%2BhUKGK3PGKwcKqzoosamn36YW-fsuTdOPPF1i_rtEO%3DnEYKSg%40mail.gmail.com RecoveryConflictInterrupt() is unsafe in a signal handler]<br /> ** This seems to [https://www.postgresql.org/message-id/447238.1651082925%40sss.pgh.pa.us explain buildfarm failures in 031_recovery_conflict.pl]<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/CAH2-WzkjjCoq5Y4LeeHJcjYJVxGm3M3SAWZ0%3D6J8K1FPSC9K0w%40mail.gmail.com REINDEX on a system catalog can leave index with two index tuples whose heap TIDs match]<br /> ** In other words, there is a rare case where the HOT invariant is violated. Same HOT chain is indexed twice due to confusion about which precise heap tuple should be indexed.<br /> ** Unclear what the user impact is.<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com memory leak with JIT inlining]<br /> ** [https://www.postgresql.org/message-id/flat/20210331040751.GU4431%40telsasoft.com#cc34872765add8e483e05009212d9d39 Another report of (same?) issue and reproducer] [https://www.postgresql.org/message-id/flat/9f73e655-14b8-feaf-bd66-c0f506224b9e%40stephans-server.de Another report] [https://www.postgresql.org/message-id/flat/16707-f5df308978a55bf8%40postgresql.org Another report] [https://www.postgresql.org/message-id/flat/CAPH-tTxLf44s3CvUUtQpkDr1D8Hxqc2NGDzGXS1ODsfiJ6WSqA%40mail.gmail.com Another report] [https://www.postgresql.org/message-id/flat/a53cacb0-8835-57d6-31e4-4c5ef196de1a@deepbluecap.com Another report]<br /> <br /> * [https://www.postgresql.org/message-id/flat/dc9dd229-ed30-6c62-4c41-d733ffff776b%40xs4all.nl TOAST fetches could perhaps occur after the needed data has been removed]<br /> ** The symptom originally reported in the thread was fixed by {{PgCommitURL|9f4f0a0dad4c7422a97d94e4051c08ec6d181dd6}}, but nobody is very happy with the status quo in this area. Do we need to do more now?<br /> ** Affects all stable branches.<br /> <br /> * [https://www.postgresql.org/message-id/ZArVOMifjzE7f8W7%40paquier.xyz Requiring recovery.signal or standby.signal when recovering with a backup_label]<br /> ** This is a rather old behavior that affects all stable branches, still not something that should be backpatched as-is.<br /> <br /> * {{messageLink|cfcca574-6967-c5ab-7dc3-2c82b6723b99@mail.ru|pg_visibility's pg_check_visible() yields false positive when working in parallel with autovacuum}}<br /> ** {{messageLink|1649062270.289865713@f403.i.mail.ru|Thread with patch}} [https://commitfest.postgresql.org/43/3739/ CF Entry]<br /> <br /> * {{messageLink|17862-1ab8f74b0f7b0611@postgresql.org|WindowAgg startup costs don't take into account partition bound. Can lead to incorrect use of cheap startup plans}}<br /> ** {{messageLink|CAApHDvrB0S5BMv+0-wTTqWFE-BJ0noWqTnDu9QQfjZ2VSpLv_g@mail.gmail.com|Patch to fix and discussion}}<br /> <br /> * {{messageLink|1516594.1681482708@sss.pgh.pa.us|We are not compatible with newly-released LLVM 16}}<br /> <br /> * {{messageLink|20230314174521.74jl6ffqsee5mtug%40awork3.anarazel.de|DROP DATABASE is interruptible}}<br /> ** Additional discussion: {{messageLink|01020187577238cf-da8c0f4a-3ab9-445a-8c74-31ef51439f30-000000%40eu-west-1.amazonses.com|&quot;PANIC: could not open critical system index 2662&quot; - twice}}<br /> <br /> === Fixed issues ===<br /> <br /> * [https://www.postgresql.org/message-id/CAEze2WgGiw%2BLZt%2BvHf8tWqB_6VxeLsMeoAuod0N%3Dij1q17n5pw%40mail.gmail.com Non-replayable WAL records through overflows and &gt;MaxAllocSize lengths]<br /> ** In other words; we can write xlog records that we can't read (plus potentially actual WAL corruption); making the instance unrecoverable, and blocks any replication.<br /> ** Exploitation seems limited to WAL records of 2PC and logical replication, and extension-generated WAL.<br /> ** Affects all stable branches.<br /> ** Fixed at: {{PgCommitURL|8fcb32db98eda1ad2a0c0b40b1cbb5d9a7aa68f0}} and {{PgCommitURL|ffd1b6bb6f8a2ffc929699772610c6925364dbb3}} for HEAD.<br /> <br /> * [https://www.postgresql.org/message-id/flat/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Fix fseek() detection of unseekable files on WIN32]<br /> ** Fixed at: {{PgCommitURL|a923e21631a29dc8b8781d7d02b5003d0df64ca3}} and {{PgCommitURL|765f5df726918bcdcfd16bcc5418e48663d1dd59}}, down to 14.<br /> <br /> * {{messageLink|CAAKRu_bETD%2BAri600h6fRjX2p8rJSeMAUp%3D_y88juqOZgouTSg%40mail.gmail.com|Can't disable autovacuum cost delay through storage parameter}}<br /> ** Fixed at: {{PgCommitURL|bfac8f8bc4a44c67c9f35b5266676278e4ba1217}}, down to 11.<br /> <br /> == Non-bugs ==<br /> <br /> == Resolved Issues ==<br /> <br /> === resolved before 16beta1 ===<br /> <br /> * Revert {{PgCommitURL|ec386948948}}, per {{messageLink|20230330105325.y6uvpalspynf2frt@alvherre.pgsql|Re: &quot;variable not found in subplan target list&quot;}}<br /> ** Reverted at {{PgCommitURL|5472743d9e8}}<br /> <br /> * [https://www.postgresql.org/message-id/CAEZATCWETioXs5kY8vT6BVguY41_wD962VDk%3Du_Nvd7S1UXzuQ%40mail.gmail.com ERROR: ORDER/GROUP BY expression not found in targetlist]<br /> ** Fixed at: {{PgCommitURL|da5800d5fa636c6e10c9c98402d872c76aa1c8d0}}<br /> <br /> * [https://www.postgresql.org/message-id/20230212233711.GA1316@telsasoft.com various elogs hit by sqlsmith (ExecRTCheckPerms() and many prunable partitions)]<br /> ** Fixed at: {{PgCommitURL|c7468c73f7b6e842a53c12eaee5578a76a8fa7a6}}<br /> <br /> * [https://www.postgresql.org/message-id/20230228235834.GC30529@telsasoft.com pg_dump: zlib compression fails for empty objects (LOs)]<br /> ** Fixed at: {{PgCommitURL|00d9dcf5bebbb355152a60f0e2120cdf7f9e7ddd}}<br /> <br /> * [https://www.postgresql.org/message-id/20230227044910.GO1653@telsasoft.com pg_dump: lz4 compression uses no persistent state and writes a block header for every row]<br /> ** Fixed at: {{PgCommitURL|0070b66fef21e909adb283f7faa7b1978836ad75}}<br /> <br /> * {{messageLink|3590249.1680971629@sss.pgh.pa.us|Assertion failure with parallel full hash join}}<br /> ** Fixed at: {{PgCommitURL|b37d051b0e59e4324e346655a27509507813db79}}<br /> <br /> * {{messageLink|ZDDO6jaESKaBgej0@tamriel.snowman.net|De-revert &quot;Add support for Kerberos credential delegation&quot;}}<br /> ** Owner: Stephen Frost<br /> ** Original commit: {{PgCommitURL|3d4fa227bce4294ce1cc214b4a9d3b7caa3f0454}}<br /> ** Revert: ({{PgCommitURL|3d03b24c350ab060bb223623bdff38835bd7afd0}}<br /> ** De-Revert: {{PgCommitURL|6633cfb21691840c33816a6dacaca0b504efb895}}<br /> ** Resolved at: {{PgCommitURL|f7431bca8b0138bdbce7025871560d39119565a0}}<br /> <br /> * {{messageLink|c39be3c5-c1a5-1e33-1024-16f527e251a4@enterprisedb.com|SSL tests break on non-existing system CA pool}}<br /> ** Fixed at: {{PgCommitURL|0b5d1fb36adda612bd3d5d032463a6eeb0729237}}<br /> <br /> * {{messageLink|CAD21AoBS7o6Ljt_vfqPQPf67AhzKu3fR0iqk8B%3DvVYczMugKMQ%40mail.gmail.com|VacuumUpdateCosts() logging condition incorrect for some initial values of vacuum_cost_delay}}<br /> ** Fixed at: {{PgCommitURL|a9781ae11ba2fdb44a3a72c9a7ebb727140b25c5}}<br /> <br /> * {{messageLink|CA%2BhUKGJ-ZPJwKHVLbqye92-ZXeLoCHu5wJL6L6HhNP7FkJ%3DmeA%40mail.gmail.com|check_strxfrm_bug()}}<br /> ** Owner: Thomas Munro<br /> ** Fixed at: {{PgCommitURL|7d3d72b55edd1b7552a9a358991555994efab0e9}}<br /> <br /> * {{messageLink|20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de|Should we remove vacuum_defer_cleanup_age?}}<br /> ** Owner: Andres Freund<br /> ** Fixed at: {{PgCommitURL|1118cd37eb61e6a2428f457a8b2026a7bb3f801a}}<br /> <br /> * {{messageLink|2fefa454-5a70-2174-ddbf-4a0e41537139@gmail.com|Add two missing tests in 035_standby_logical_decoding.pl}}<br /> ** Fixed at: {{PgCommitURL|376dc820531bafcbf105fff74c5b14c23d9950af}}<br /> ** Fixed at: {{PgCommitURL|a6e04b1d20c2e9cece9b64bb5b36ebfdc3a9031b}}<br /> <br /> == Won't Fix ==<br /> <br /> == Important Dates ==<br /> <br /> Current schedule:<br /> <br /> * Beta 2: TBD<br /> * Beta 1: May 25, 2023<br /> * Feature Freeze: April 8, 2023 0:00 AoE ('''Last Day to Commit Features''')<br /> <br /> == See also ==<br /> <br /> * [[Release Management Team]]<br /> * [[PostgreSQL 15 Open Items]]<br /> <br /> [[Category:Open_Items]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=PgCon_2023_Developer_Meeting&diff=37681 2023-03-21T21:38:03Z

<p>Andresfreund: /* RSVPs */</p> <hr /> <div>A meeting of the interested PostgreSQL developers is being planned for Tuesday 30 May, 2023 at the University of Ottawa, prior to pgCon 2023. In order to keep the numbers manageable, this meeting is by '''invitation only'''.<br /> Any questions regarding the invitations to this event should be directed to the team of individuals tasked with coming up with the list of people to invite:<br /> <br /> * Andres Freund<br /> * Stephen Frost<br /> * Dave Page<br /> <br /> An Unconference will be held on Friday for in-depth discussion of technical topics.<br /> <br /> This is a PostgreSQL Community event.<br /> <br /> == Meeting Goals ==<br /> <br /> * Define the schedule for the upcoming releases<br /> * Address any proposed timing, policy, or procedure issues<br /> * Receive updates from project sub-teams on their activities and discuss any resulting issues or concerns.<br /> * Address any proposed [http://en.wikipedia.org/wiki/Wicked_problem Wicked problems]<br /> <br /> == Time &amp; Location ==<br /> <br /> The meeting will (probably) be:<br /> <br /> * 9:00AM to 12PM<br /> * DMS 3105 - Desmarais Hall, 55 Laurier Avenue East<br /> * University of Ottawa.<br /> <br /> Lunch will be served during the meeting.<br /> <br /> == COVID-19 ==<br /> <br /> The University of Ottawa's COVID-19 guidance can be found at https://www.uottawa.ca/en/covid-19. Wearing of masks at the Developer Meeting will be optional, however we do ask that people do not attend if they have COVID symptoms or have tested positive.<br /> <br /> == RSVPs ==<br /> <br /> The following people have RSVPed to the meeting (in alphabetical order, by surname). Note that we can accommodate a '''maximum of 30'''!<br /> <br /> # Joe Conway<br /> # Jeff Davis<br /> # Stephen Frost<br /> # Jonathan Katz<br /> # Alexander Korotkov<br /> # Tom Lane<br /> # Dave Page<br /> # Andres Freund<br /> <br /> The following people will not be in Ottawa, and do not plan to attend:<br /> <br /> == Agenda Items ==<br /> <br /> * 16.0 release and commitfest schedule (Dave)<br /> * ''Please add suggestions for agenda items here. (with your name)''<br /> <br /> ==Agenda==<br /> <br /> {| border=&quot;1&quot; cellpadding=&quot;4&quot; cellspacing=&quot;0&quot;<br /> !Time<br /> !Item<br /> !Presenter<br /> <br /> |- style=&quot;font-style:italic;background-color:lightgray;&quot;<br /> |09:00 - 09:10<br /> |Welcome and introductions<br /> |Dave Page<br /> <br /> |- <br /> |09:10 - 09:20<br /> |Release and commitfest schedules<br /> |Dave Page<br /> <br /> |- <br /> |??:?? - ??:??<br /> |TBD<br /> |TBD<br /> <br /> |- style=&quot;font-style:italic;background-color:lightgray;&quot;<br /> |10:30 - 11:00<br /> |Coffee break<br /> |All<br /> <br /> |- <br /> |??:?? - ??:??<br /> |TBD<br /> |TBD<br /> <br /> |- <br /> |11:50 - 12:00<br /> |Any other business<br /> |Dave Page<br /> <br /> |- style=&quot;font-style:italic;background-color:lightgray;&quot;<br /> |12:00<br /> |Lunch<br /> |<br /> <br /> |}<br /> <br /> Note: This timetable is a rough guide only. Items will start as soon as the previous discussion is complete (breaks will not move materially however). Any remaining time before lunch may be used for Commitfest item triage or other activities.<br /> <br /> [[Category:Developer Meeting]]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37274 2022-10-05T22:07:59Z

<p>Andresfreund: /* Installing Meson */ - Add RHEL n</p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> || CC is only checked during meson setup, not with meson configure<br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || install all the binaries and libraries<br /> || make install<br /> || ninja install<br /> || use meson install --quiet for a less verbose experience<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || build documentation<br /> || cd doc/ &amp;&amp; make html &amp;&amp; make man<br /> || ninja docs<br /> || Builds html documentation and man pages <br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> ||<br /> |-<br /> || list tests<br /> ||<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9 (tested on Rocky Linux 9):<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled crb<br /> dnf -y install meson<br /> &lt;/pre&gt;<br /> <br /> === macOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> <br /> On arm64:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> On x86-64:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> == Why and What ==<br /> <br /> Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br /> it. Recursive make has a lot of hard to resolve dependency issues and slow<br /> incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br /> developers not using windows and runs tests serially. While these and other<br /> issues could individually be addressed with incremental improvements, together<br /> they seem best addressed by moving to a more modern buildsystem.<br /> <br /> After evaluating different buildsystem choices, we chose to use meson, to a<br /> good degree based on the adoption by other open source projects.<br /> <br /> We decided that it's more realistic to commit a relatively early version of<br /> the new buildsystem and mature it in tree.<br /> <br /> The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br /> after reaching feature parity. However, we're not planning to remove the<br /> autoconf/make buildsystem in the near future. Likely we're going to keep at<br /> least the parts required for PGXS to keep working around until all supported<br /> versions build with meson.<br /> <br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br /> <br /> <br /> == Development tree, other resources ==<br /> <br /> * https://github.com/anarazel/postgres/tree/meson<br /> * https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br /> <br /> == Visualizing builds ==<br /> <br /> When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37241 2022-09-26T19:02:38Z

<p>Andresfreund: Blind adjustment for macOS homebrew on 86_64, reported by John Naylor</p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> || CC is only checked during meson setup, not with meson configure<br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || install all the binaries and libraries<br /> || make install<br /> || ninja install<br /> || use meson install --quiet for a less verbose experience<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || build documentation<br /> || cd doc/ &amp;&amp; make html &amp;&amp; make man<br /> || ninja docs<br /> || Builds html documentation and man pages <br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> ||<br /> |-<br /> || list tests<br /> ||<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === macOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> <br /> On arm64:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig -Dextra_include_dirs=/opt/homebrew/include -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> On x86-64:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/usr/local/lib/pkgconfig -Dextra_include_dirs=/usr/local/include -Dextra_lib_dirs=/usr/local/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> <br /> == Why and What ==<br /> <br /> Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br /> it. Recursive make has a lot of hard to resolve dependency issues and slow<br /> incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br /> developers not using windows and runs tests serially. While these and other<br /> issues could individually be addressed with incremental improvements, together<br /> they seem best addressed by moving to a more modern buildsystem.<br /> <br /> After evaluating different buildsystem choices, we chose to use meson, to a<br /> good degree based on the adoption by other open source projects.<br /> <br /> We decided that it's more realistic to commit a relatively early version of<br /> the new buildsystem and mature it in tree.<br /> <br /> The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br /> after reaching feature parity. However, we're not planning to remove the<br /> autoconf/make buildsystem in the near future. Likely we're going to keep at<br /> least the parts required for PGXS to keep working around until all supported<br /> versions build with meson.<br /> <br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br /> <br /> <br /> == Development tree, other resources ==<br /> <br /> * https://github.com/anarazel/postgres/tree/meson<br /> * https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br /> <br /> == Visualizing builds ==<br /> <br /> When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37240 2022-09-26T18:59:02Z

<p>Andresfreund: mention &quot;meson install --quiet&quot; instead of copying comment about parallelism</p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> || CC is only checked during meson setup, not with meson configure<br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || install all the binaries and libraries<br /> || make install<br /> || ninja install<br /> || use meson install --quiet for a less verbose experience<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || build documentation<br /> || cd doc/ &amp;&amp; make html &amp;&amp; make man<br /> || ninja docs<br /> || Builds html documentation and man pages <br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> ||<br /> |-<br /> || list tests<br /> ||<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig/ -Dextra_include_dirs=/opt/homebrew/include/ -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> <br /> == Why and What ==<br /> <br /> Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br /> it. Recursive make has a lot of hard to resolve dependency issues and slow<br /> incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br /> developers not using windows and runs tests serially. While these and other<br /> issues could individually be addressed with incremental improvements, together<br /> they seem best addressed by moving to a more modern buildsystem.<br /> <br /> After evaluating different buildsystem choices, we chose to use meson, to a<br /> good degree based on the adoption by other open source projects.<br /> <br /> We decided that it's more realistic to commit a relatively early version of<br /> the new buildsystem and mature it in tree.<br /> <br /> The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br /> after reaching feature parity. However, we're not planning to remove the<br /> autoconf/make buildsystem in the near future. Likely we're going to keep at<br /> least the parts required for PGXS to keep working around until all supported<br /> versions build with meson.<br /> <br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br /> <br /> <br /> == Development tree, other resources ==<br /> <br /> * https://github.com/anarazel/postgres/tree/meson<br /> * https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br /> <br /> == Visualizing builds ==<br /> <br /> When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37239 2022-09-26T18:56:50Z

<p>Andresfreund: Remove unnecessary use of -v for ninja install, noticed by John Naylor</p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> || CC is only checked during meson setup, not with meson configure<br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || install all the binaries and libraries<br /> || make install<br /> || ninja install<br /> || ninja uses parallelism by default, launch from the root of the build tree.<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || build documentation<br /> || cd doc/ &amp;&amp; make html &amp;&amp; make man<br /> || ninja docs<br /> || Builds html documentation and man pages <br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> ||<br /> |-<br /> || list tests<br /> ||<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig/ -Dextra_include_dirs=/opt/homebrew/include/ -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> <br /> == Why and What ==<br /> <br /> Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br /> it. Recursive make has a lot of hard to resolve dependency issues and slow<br /> incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br /> developers not using windows and runs tests serially. While these and other<br /> issues could individually be addressed with incremental improvements, together<br /> they seem best addressed by moving to a more modern buildsystem.<br /> <br /> After evaluating different buildsystem choices, we chose to use meson, to a<br /> good degree based on the adoption by other open source projects.<br /> <br /> We decided that it's more realistic to commit a relatively early version of<br /> the new buildsystem and mature it in tree.<br /> <br /> The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br /> after reaching feature parity. However, we're not planning to remove the<br /> autoconf/make buildsystem in the near future. Likely we're going to keep at<br /> least the parts required for PGXS to keep working around until all supported<br /> versions build with meson.<br /> <br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br /> <br /> <br /> == Development tree, other resources ==<br /> <br /> * https://github.com/anarazel/postgres/tree/meson<br /> * https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br /> <br /> == Visualizing builds ==<br /> <br /> When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37233 2022-09-22T06:15:21Z

<p>Andresfreund: </p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> || CC is only checked during meson setup, not with meson configure<br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> |-<br /> || list tests<br /> |<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig/ -Dextra_include_dirs=/opt/homebrew/include/ -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> <br /> == Why and What ==<br /> <br /> Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br /> it. Recursive make has a lot of hard to resolve dependency issues and slow<br /> incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br /> developers not using windows and runs tests serially. While these and other<br /> issues could individually be addressed with incremental improvements, together<br /> they seem best addressed by moving to a more modern buildsystem.<br /> <br /> After evaluating different buildsystem choices, we chose to use meson, to a<br /> good degree based on the adoption by other open source projects.<br /> <br /> We decided that it's more realistic to commit a relatively early version of<br /> the new buildsystem and mature it in tree.<br /> <br /> The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br /> after reaching feature parity. However, we're not planning to remove the<br /> autoconf/make buildsystem in the near future. Likely we're going to keep at<br /> least the parts required for PGXS to keep working around until all supported<br /> versions build with meson.<br /> <br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br /> <br /> <br /> == Development tree, other resources ==<br /> <br /> * https://github.com/anarazel/postgres/tree/meson<br /> * https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal<br /> <br /> == Visualizing builds ==<br /> <br /> When building with ninja, the generated .ninja_log can be uploaded to [https://ui.perfetto.dev/ ui.perfetto.dev], which is very helpful to visualize builds.</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37230 2022-09-21T16:20:23Z

<p>Andresfreund: </p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> || CC is only checked during meson setup, not with meson configure<br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> |-<br /> || list tests<br /> |<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig/ -Dextra_include_dirs=/opt/homebrew/include/ -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> <br /> == Why and What ==<br /> <br /> Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br /> it. Recursive make has a lot of hard to resolve dependency issues and slow<br /> incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br /> developers not using windows and runs tests serially. While these and other<br /> issues could individually be addressed with incremental improvements, together<br /> they seem best addressed by moving to a more modern buildsystem.<br /> <br /> After evaluating different buildsystem choices, we chose to use meson, to a<br /> good degree based on the adoption by other open source projects.<br /> <br /> We decided that it's more realistic to commit a relatively early version of<br /> the new buildsystem and mature it in tree.<br /> <br /> The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br /> after reaching feature parity. However, we're not planning to remove the<br /> autoconf/make buildsystem in the near future. Likely we're going to keep at<br /> least the parts required for PGXS to keep working around until all supported<br /> versions build with meson.<br /> <br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br /> <br /> <br /> == Development tree, other resources ==<br /> <br /> * https://github.com/anarazel/postgres/tree/meson<br /> * https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37229 2022-09-21T16:19:00Z

<p>Andresfreund: </p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> |-<br /> || list tests<br /> |<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig/ -Dextra_include_dirs=/opt/homebrew/include/ -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> <br /> == Why and What ==<br /> <br /> Autoconf is showing its age, fewer and fewer contributors know how to wrangle<br /> it. Recursive make has a lot of hard to resolve dependency issues and slow<br /> incremental rebuilds. Our home-grown MSVC buildsystem is hard to maintain for<br /> developers not using windows and runs tests serially. While these and other<br /> issues could individually be addressed with incremental improvements, together<br /> they seem best addressed by moving to a more modern buildsystem.<br /> <br /> After evaluating different buildsystem choices, we chose to use meson, to a<br /> good degree based on the adoption by other open source projects.<br /> <br /> We decided that it's more realistic to commit a relatively early version of<br /> the new buildsystem and mature it in tree.<br /> <br /> The plan is to remove the msvc specific buildsystem in src/tools/msvc soon<br /> after reaching feature parity. However, we're not planning to remove the<br /> autoconf/make buildsystem in the near future. Likely we're going to keep at<br /> least the parts required for PGXS to keep working around until all supported<br /> versions build with meson.<br /> <br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]<br /> <br /> <br /> == Development tree, other resources ==<br /> <br /> * https://github.com/anarazel/postgres/tree/meson<br /> * https://wiki.postgresql.org/wiki/PgCon_2022_Developer_Unconference#Meson_new_build_system_proposal</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37228 2022-09-21T05:14:23Z

<p>Andresfreund: </p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt; <br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] [&lt;build dir&gt;] &lt;source-dir&gt;<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> |-<br /> || list tests<br /> |<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find Homebrew libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig/ -Dextra_include_dirs=/opt/homebrew/include/ -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37227 2022-09-21T00:08:51Z

<p>Andresfreund: </p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] &lt;source-dir&gt; [&lt;build dir&gt;]<br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] &lt;source-dir&gt; [&lt;build dir&gt;]<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> |-<br /> || list tests<br /> |<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/homebrew/lib/pkgconfig/ -Dextra_include_dirs=/opt/homebrew/include/ -Dextra_lib_dirs=/opt/homebrew/lib $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37226 2022-09-20T23:20:19Z

<p>Andresfreund: </p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] &lt;source-dir&gt; [&lt;build dir&gt;]<br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] &lt;source-dir&gt; [&lt;build dir&gt;]<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> |-<br /> || list tests<br /> |<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> <br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> <br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> <br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> <br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> Arguments to meson setup/configure to find MacPorts libraries:<br /> &lt;pre&gt;<br /> meson setup -Dpkg_config_path=/opt/local/lib/pkgconfig -Dextra_lib_dirs=/opt/local/lib/ -Dextra_include_dirs=/opt/local/include $builddir $sourcedir<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> <br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> <br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]</div>

Andresfreund https://wiki.postgresql.org/index.php?title=Meson&diff=37225 2022-09-20T22:52:31Z

<p>Andresfreund: </p> <hr /> <div>== Quickstart ==<br /> <br /> {|class=&quot;wikitable&quot; style=&quot;margin:auto&quot;<br /> |+ translation of configure, make, etc commands <br /> !description<br /> !old command<br /> !new command<br /> !comment<br /> |-<br /> || set up build tree<br /> || ./configure [&lt;options&gt;]<br /> || meson setup [&lt;options&gt;] &lt;source-dir&gt; [&lt;build dir&gt;]<br /> || meson only supports building out of tree<br /> |-<br /> || set up build tree for visual studio<br /> || perl src/tools/msvc/mkvcbuild.pl<br /> || meson setup --backend vs [&lt;options&gt;] &lt;source-dir&gt; [&lt;build dir&gt;]<br /> || configures build tree for one build type (debug or release or ...)<br /> |-<br /> || show configure options<br /> || ./configure --help<br /> || meson configure<br /> || shows options built into meson and PostgreSQL specific options<br /> |-<br /> || set configure options<br /> || ./configure --prefix=DIR, --$somedir=DIR, --with-$option, --enable-$feature<br /> || meson setup|configure -D$option=$value<br /> || options can be set when setting up build tree (setup) and in existing build tree (configure)<br /> |- <br /> || enable cassert<br /> || --enable-cassert<br /> || -Dcassert=true<br /> |-<br /> || enable debug symbols<br /> || ./configure --enable-debug<br /> || meson configure|setup -Ddebug=true<br /> |-<br /> || specify compiler<br /> || CC=compiler ./configure<br /> || CC=compiler meson setup <br /> |-<br /> || set CFLAGS<br /> || CFLAGS=options ./configure<br /> || meson configure|setup -Dc_args=options<br /> || CFLAGS is also checked, but only for meson setup<br /> |-<br /> || build<br /> || make -s<br /> || ninja<br /> || ninja uses parallelism by default<br /> |-<br /> || build, showing compiler commands<br /> || make<br /> || ninja -v<br /> || ninja uses parallelism by default<br /> |-<br /> || run all tests<br /> || make check-world<br /> || meson test<br /> || runs all test, using parallelism by default<br /> |-<br /> || run main regression tests<br /> || make check<br /> || meson test --suite setup --suite main<br /> |-<br /> || list tests<br /> |<br /> || meson test --list<br /> |}<br /> <br /> == Installing Meson ==<br /> <br /> === Linux ===<br /> <br /> Debian / Ubuntu:<br /> <br /> &lt;pre&gt;<br /> apt-get update &amp;&amp; apt-get install -y meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> Fedora:<br /> <br /> &lt;pre&gt;<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 8:<br /> <br /> &lt;pre&gt;<br /> dnf -y install dnf-plugins-core<br /> dnf config-manager --set-enabled powertools<br /> dnf -y install meson ninja-build<br /> &lt;/pre&gt;<br /> <br /> RHEL 9:<br /> <br /> &lt;pre&gt;<br /> ???<br /> &lt;/pre&gt;<br /> <br /> === MacOS ===<br /> <br /> With MacPorts:<br /> <br /> &lt;pre&gt;<br /> sudo port install meson<br /> &lt;/pre&gt;<br /> <br /> With Homebrew:<br /> <br /> &lt;pre&gt;<br /> brew install meson<br /> &lt;/pre&gt;<br /> <br /> === Windows ===<br /> <br /> Assuming python is installed, the easiest way to get meson and ninja is:<br /> <br /> &lt;pre&gt;<br /> pip install meson ninja<br /> &lt;/pre&gt;<br /> <br /> == Meson documentation ==<br /> <br /> * [https://mesonbuild.com/Commands.html meson commandline commands]<br /> * [https://mesonbuild.com/Syntax.html meson syntax]<br /> * [https://mesonbuild.com/Reference-manual_functions.html meson functions]</div>

Andresfreund