Todo
From PostgreSQL wiki
(Redirected from XML Todo)
Jump to navigationJump to searchThis list contains some known PostgreSQL bugs, some feature requests, and some things we are not even sure we want. Many of these items are hard, and some are perhaps impossible. If you would like to work on an item, please read the Developer FAQ first. There is also a development information page.
- marks incomplete items- [D]
- marks changes that are done, and will appear in the PostgreSQL 17 release.
Over time, it may become clear that a TODO item has become outdated or otherwise determined to be either too controversial or not worth the development effort. Such items should be retired to the Not Worth Doing page.
For help on editing this list, please see Talk:Todo. Please do not add or remove items here without discussion on the mailing list.
Development Process
WARNING for Developers: Unfortunately this list does not contain all the information necessary for someone to start coding a feature. Some of these items might have become unnecessary since they were added --- others might be desirable but the implementation might be unclear. When selecting items listed below, be prepared to first discuss the value of the feature. Do not assume that you can select one, code it and then expect it to be committed. Always discuss design on Hackers list before starting to code. The flow should be:
Desirability -> Design -> Implement -> Test -> Review -> Commit
Administration
- Check for unreferenced table files created by transactions that were in-progress when the server terminated abruptly
- Allow log_min_messages to be specified on a per-module basis
- This would allow administrators to see more detailed information from specific sections of the backend, e.g. checkpoints, autovacuum, etc. Another idea is to allow separate configuration files for each module, or allow arbitrary SET commands to be passed to them. See also Logging Brainstorm.
- Prevent query cancel packets from being replayed by an attacker, especially when using SSL
- Maintain an approximate xid->"time of assignment" mapping. This would allow controlling the maximum effect of hot_standby_feedback on the primary, and would be useful for other features.
Tablespaces
- Allow WAL replay of CREATE TABLESPACE to work when the directory structure on the recovery computer is different from the original
- Allow tablespaces on RAM-based partitions for unlogged tables
- Allow toast tables to be moved to a different tablespace
Statistics Collector
- Testing pgstat via pg_regress is tricky and inefficient. Consider making a dedicated pgstat test-suite.
- Teach stats collector to differentiate between internal and leaf index pages
SSL
- Allow automatic selection of SSL client certificates from a certificate store
Standby server mode
- Prevent variables inherited from the server environment from being used for making streaming replication connections
Data Types
- Add support for public SYNONYMs
- Consider a special data type for regular expressions
- Allow deleting enumerated values from an existing enumerated data type
- Add overlaps geometric operators that ignore point overlaps
Arrays
- Allow single-byte header storage for array elements
Text Search
- Consider changing error to warning for strings larger than one megabyte
- Improve default parser, to more easily allow adding new tokens
- Add additional support functions
XML
- Add XML Schema validation and xmlvalidate functions (SQL:2008)
- Add xmlvalidatedtd variant to support validating against a DTD?
- Relax-NG validation; libxml2 supports this already
- Add functions from SQL:2006: XMLDOCUMENT, XMLCAST, XMLTEXT
- XMLDocument: https://commitfest.postgresql.org/51/5431/
- XMLCast: https://commitfest.postgresql.org/50/5110/
- XMLText: https://commitfest.postgresql.org/43/4257/
- Add XMLNAMESPACES support in XMLELEMENT and elsewhere
- Report errors returned by the XSLT library
- XML Canonical: Convert XML documents to canonical form to compare them. libxml2 has support for this.
- Add pretty-printed XML output option
- Parse a document and serialize it back in some indented form. libxml2 might support this.
- Add XMLQUERY (from the SQL/XML standard)
- Allow XML shredding
- In some cases shredding could be better option (if there is no need to keep XML docs entirely, e.g. if we have already developed tools that understand only relational data. This would be a separate module that implements annotated schema decomposition technique, similar to DB2 and SQL Server functionality.
- Verify Xpath escaping behavior
Functions
- Implement Boyer-Moore searching in LIKE queries
- Prevent malicious functions from being executed with the permissions of unsuspecting users
- Indexed functions are safe, so VACUUM and ANALYZE are safe too. Triggers, CHECK and DEFAULT expressions, and rules are still vulnerable.
- Fix /contrib/btree_gist's implementation of inet indexing
- Add array_sort function
- Probably beginner-friendly, given that we already have array_shuffle and intarray::sort to reference. Plus most everything is order-able due to btree indexes.
Multi-Language Support
- Add NCHAR (as distinguished from ordinary varchar)
- Integrate collations with text search configurations
- Integrate collations with to_char() and related functions
- Support collation-sensitive equality and hashing functions
- Support multiple simultaneous character sets, per SQL:2008
- Fix contrib/fuzzystrmatch to work with multibyte encodings
- Change memory allocation for multi-byte functions so memory is allocated inside conversion functions
- Currently we preallocate memory based on worst-case usage.
- Add ability to use case-insensitive regular expressions on multi-byte characters
- Currently it works for UTF-8, but not other multi-byte encodings
- Improve encoding of connection startup messages sent to the client
- Currently some authentication error messages are sent in the server encoding
- Windows: Cache MessageEncoding conversion for use outside transactions
Views and Rules
- Improve ability to modify views via ALTER TABLE
SQL Commands
- Add CORRESPONDING BY to UNION/INTERSECT/EXCEPT
- Improve type determination of unknown (NULL or quoted literal) result columns for UNION/INTERSECT/EXCEPT
- Allow prepared transactions with temporary tables created and dropped in the same transaction, and when an ON COMMIT DELETE ROWS temporary table is accessed
- Allow NOTIFY in rules involving conditionals
- Allow NOTIFY to handle transaction wraparound
- Allow LISTEN on patterns
- Simplify dropping roles that have objects in several databases
- Add support for WITH RECURSIVE ... CYCLE
- Add DEFAULT .. AS OWNER so permission checks are done as the table owner
- This would be useful for SERIAL nextval() calls and CHECK constraints.
- Add comments on system tables/columns using the information in catalogs.sgml
- Ideally the information would be pulled from the SGML file automatically.
- Prevent the specification of conflicting transaction read/write options
- Allow PREPARE of cursors
- Have DISCARD PLANS discard plans cached by functions
- DISCARD ALL should do the same.
- Avoid multiple-evaluation of BETWEEN and IN arguments containing volatile expressions
CREATE
- Allow CREATE TABLE AS to determine column lengths for complex expressions like SELECT col1
- Have WITH CONSTRAINTS also create constraint indexes
- Move NOT NULL constraint information to pg_constraint
-
Currently NOT NULL constraints are stored in pg_attribute without any designation of their origins, e.g. primary keys. One manifest problem is that dropping a PRIMARY KEY constraint does not remove the NOT NULL constraint designation. Another issue is that we should probably force NOT NULL to be propagated from parent tables to children, just as CHECK constraints are. (But then does dropping PRIMARY KEY affect children?)
- http://archives.postgresql.org/message-id/19768.1238680878@sss.pgh.pa.us
- http://archives.postgresql.org/message-id/200909181005.n8IA5Ris061239@wwwmaster.postgresql.org
- http://archives.postgresql.org/pgsql-hackers/2011-07/msg01223.php
- http://archives.postgresql.org/pgsql-hackers/2011-07/msg00358.php
- Prevent ALTER TABLE DROP NOT NULL on child tables if parent column has it
- Prevent concurrent CREATE TABLE from sometimes returning a cryptic error message
- Add CREATE SCHEMA ... LIKE that copies a schema
- Fix CREATE OR REPLACE FUNCTION to not leave objects depending on the function in inconsistent state
- Allow temporary tables to exist as empty by default in all sessions
- Allow the creation of "distinct" types
- Consider analyzing temporary tables when they are first used in a query
- Autovacuum cannot analyze or vacuum temporary tables.
UPDATE
- Research self-referential UPDATEs that see inconsistent row versions in read-committed mode
- Improve performance of EvalPlanQual mechanism that rechecks already-updated rows
- This is related to the previous item, which questions whether it even has the right semantics
- Add system that disables HOT and block-local updates when there are significant amounts of free space earlier in the table, and automates updating the tuples in the tail of the table.
- This moves tuples more to the front of the table, increasing the chance a subsequent VACUUM operation can truncate parts of the table.
ALTER
- Have ALTER TABLE RENAME of a SERIAL column rename the sequence
- Add ALTER DOMAIN to modify the underlying data type
- Allow ALTER TABLESPACE to move the tablespace to different directories
- Allow moving system tables to other tablespaces, where possible
- Currently non-global system tables must be in the default database tablespace. Global system tables can never be moved.
- Have ALTER INDEX update the name of a constraint using that index
- Allow column display reordering by recording a display, storage, and permanent id for every column?
- Allow deactivating (and reactivating) indexes via ALTER TABLE
- Add ALTER OPERATOR ... RENAME
- needs to consider effects of changing operator precedence
CLUSTER
- Automatically maintain clustering on a table
- This might require some background daemon to maintain clustering during periods of low usage. It might also require tables to be only partially filled for easier reorganization. Another idea would be to create a merged heap/index data file so an index lookup would automatically access the heap data too. A third idea would be to store heap rows in hashed groups, perhaps using a user-supplied hash function.
- Allow CLUSTER to be used on partial indexes
COPY
- Allow COPY to report errors sooner
- Allow COPY FROM to create index entries in bulk
- Allow a stalled COPY to exit if the backend is terminated
GRANT/REVOKE
- Allow dropping of a role that has connection rights
DECLARE CURSOR
- Prevent DROP TABLE from dropping a table referenced by its own open cursor?
- Provide some guarantees about the behavior of cursors that invoke volatile functions
SHOW/SET
- Rationalize the discrepancy between settings that use values in bytes and SHOW that returns the object count
ANALYZE
- Improve how ANALYZE computes in-doubt tuples
- Reduce memory use when analyzing many tables in a single command by making catcache and syscache flushable or bounded.
EXPLAIN
- Have EXPLAIN ANALYZE issue NOTICE messages when the estimated and actual row counts differ by a specified percentage
- Have EXPLAIN ANALYZE report rows as floating-point numbers
- Have EXPLAIN ANALYZE report buckets and memory usage for HashAggregate
Window Functions
See TODO items for window functions.
- Support creation of user-defined window functions
- We have the ability to create new window functions written in C. Is it worth the effort to create an API that would let them be written in PL/pgsql, etc?
- Investigate tuplestore performance issues
- The tuplestore_in_memory() thing is just a band-aid, we ought to try to solve it properly. tuplestore_advance seems like a weak spot as well.
- Do we really need so much duplicated code between Agg and WindowAgg?
- Teach planner to evaluate multiple windows in the optimal order
- Currently windows are always evaluated in the query-specified order.
- Implement DISTINCT clause in window aggregates
- Some proprietary RDBMSs have implemented it already, so it helps with porting from those.
Integrity Constraints
Keys
- Improve deferrable unique constraints for cases with many conflicts
- The current implementation fires a trigger for each potentially conflicting row. This might not scale well for an update that changes many key values at once.
Referential Integrity
- Add MATCH PARTIAL referential integrity
- Change foreign key constraint for array -> element to mean element in array?
- Fix problem when cascading referential triggers make changes on cascaded tables, seeing the tables in an intermediate state
- Are ri_KeysEqual checks in the RI enforcement triggers still necessary?
Check Constraints
- Run check constraints only when affected columns are changed
- Do not scan the table when a check constraint is added in the same command that adds the column
Server-Side Languages
- Add support for polymorphic arguments and return types to languages other than PL/PgSQL
- Add support for OUT and INOUT parameters to languages other than PL/PgSQL
- Add more fine-grained specification of functions taking arbitrary data types
- Allow holdable cursors in SPI
- Rethink query plan caching and timing of parse analysis within SQL-language functions
- They should work more like plpgsql functions do ...
- Allow regex operations in PL/Perl using UTF8 characters in non-UTF8 encoded databases
PL/pgSQL
- Allow listing of record column names, and access to record columns via variables, e.g. columns := r.(*), tval2 := r.(colname)
- Allow row and record variables to be set to NULL constants, and allow NULL tests on such variables
- Because a row is not scalar, do not allow assignment from NULL-valued scalars.
- Consider keeping separate cached copies when search_path changes
- Improve handling of NULL row values vs. NULL rows
- Improve PERFORM handling of WITH queries or document limitation
PL/Python
- Create a new restricted execution class that will allow passing function arguments in as locals. Passing them as globals means functions cannot be called recursively.
- Add a DB-API compliant interface on top of the SPI interface
- For functions returning a setof record with a composite type, cache the I/O functions for the composite type
Clients
- Split out pg_resetxlog output into pre- and post-sections
- Improve pg_rewind
psql
- Move psql backslash database information into the backend, use mnemonic commands?
- This would allow non-psql clients to pull the same information out of the database as psql.
- Consistently display privilege information for all objects in psql
- Add a \set variable to control whether \s displays line numbers
- Another option is to add \# which lists line numbers, and allows command execution.
- Include the symbolic SQLSTATE name in verbose error reports
- Add option to wrap column values at whitespace boundaries, rather than chopping them at a fixed width.
- Currently, "wrapped" format chops values into fixed widths. Perhaps the word wrapping could use the same algorithm documented in the W3C specification.
- Fix FETCH_COUNT to handle SELECT ... INTO and WITH queries
- Prevent psql from sending remaining single-line multi-statement queries after reconnecting
- Improve speed of tab completion by using LIKE
- Add handling of Unicode grapheme clusters
pg_dump / pg_restore
- Dump security labels and comments on databases in a way that allows to load a dump into a differently named database
- Add full object name to the tag field. eg. for operators we need '=(integer, integer)', instead of just '='.
- Avoid using platform-dependent names for locales in pg_dumpall output
- Using native locale names puts roadblocks in the way of porting a dump to another platform. One possible solution is to get CREATE DATABASE to accept some agreed-on set of locale names and fix them up to meet the platform's requirements.
- In a selective dump, allow dumping of an object and all its dependencies
- Stop dumping CASCADE on DROP TYPE commands in clean mode
- Allow pg_restore to load different parts of the COPY data for a single table simultaneously
- Preserve sparse storage of large objects over dump/restore
- Prevent PL/pgSQL comment from throwing an error in a non-superuser restore
- Delay REFRESH MATERIALIZED VIEW until dependent indexes are created
- Remove or document the behavior of --data-only
pg_upgrade
- Handle large object comments
- This is difficult to do because the large object doesn't exist when --schema-only is loaded.
- Consider using pg_depend for checking object usage in version.c
- Migrate pg_statistic by dumping it out as a flat file, so analyze is not necessary
- Find cleaner way to start/stop dedicated servers for upgrades
- Desired changes that would prevent upgrades with pg_upgrade
- 32-bit page checksums
- Add metapage to GiST indexes
- Clean up hstore's internal representation
- Remove tuple infomask bit HEAP_MOVED_OFF and HEAP_MOVED_IN
- fix char() index trailing space handling
- Use non-collation-aware comparisons for GIN opclasses
ecpg
- Docs
- Document differences between ecpg and the SQL standard and information about the Informix-compatibility module.
- Solve cardinality > 1 for input descriptors / variables?
- Add a semantic check level, e.g. check if a table exists
- fix handling of DB attributes that are arrays
- sqlwarn[6] should be 'W' if the PRECISION or SCALE value specified
- Make SET CONNECTION thread-aware, non-standard?
- Allow multidimensional arrays
- Implement COPY FROM STDIN
- Provide a way to specify size of a bytea parameter
- Allow reuse of cursor name variables
libpq
- Add PQexecf() that allows complex parameter substitution
- Add SQLSTATE and severity to errors generated within libpq itself
- Add support for interface/ipaddress binding to libpq
- When receiving a FATAL error remember it, so that it doesn't profess ingnorance about why the session was closed
- Pipelining support for libpq async API and an array-valued PQexecPrepared that uses it
Triggers
- Improve storage of deferred trigger queue
- Right now all deferred trigger information is stored in backend memory. This could exhaust memory for very large trigger queues. This item involves dumping large queues into files, or doing some kind of join to process all the triggers, some bulk operation, or a bitmap.
- With disabled triggers, allow pg_dump to use ALTER TABLE ADD FOREIGN KEY
- If the dump is known to be valid, allow foreign keys to be added without revalidating the data.
- When statement-level triggers are defined on a parent table, have them fire only on the parent table, and fire child table triggers only where appropriate
- Tighten trigger permission checks
- Allow BEFORE INSERT triggers on views
- Add database and transaction-level triggers
- Avoid requirement for AFTER trigger functions to return a value
- Allow creation of inline triggers
Inheritance
- Allow unique indexes across inherited tables (requires multi-table indexes)
- Postgres 11 allows unique indexes across partitions if the partition key is part of the index.
- Research whether ALTER TABLE / SET SCHEMA should work on inheritance hierarchies (and thus support ONLY)
- ALTER TABLE variants sometimes support recursion and sometimes not, but this is poorly/not documented, and the ONLY marker would then be silently ignored. Clarify the documentation, and reject ONLY if it is not supported.
Indexes
- Prevent index uniqueness checks when UPDATE does not modify the column
- Uniqueness (index) checks are done when updating a column even if the column is not modified by the UPDATE. However, HOT already short-circuits this in common cases, so more work might not be helpful.
- Consider having a larger statistics target for indexed columns and expression indexes.
- Allow multiple indexes to be created concurrently, ideally via a single heap scan
- pg_restore allows parallel index builds, but it is done via subprocesses, and there is no SQL interface for this. Cluster could definitely benefit from this.
- Consider sorting entries before inserting into btree index
- Consider using "effective_io_concurrency" for index scans
- Currently only bitmap scans use this, which might be fine because most multi-row index scans use bitmap scans.
- Allow GIN indexes to be used for exclusion constraints
- Allow "loose" scans on btree indexes in which the first column has low cardinality
- Re: Loose Index Scans by Planner
- On -hackers, but using the term "skip scan" instead of "loose index scan": Index Skip Scan, and Index Skip Scan (new UniqueKeys)
- Allow "skip" scans on multi-column btree indexes
This means applying techniques similar to those detailed in the Multidimensional Access Method (MDAM) paper
- Make the planner's "special index operator" mechanism extensible
- Allow index-only COUNT(*) for indexes which don't support index-only scans
- Improve GIN performance
- Teach GIN cost estimation about "fast scans"
- Allow unlogged indexes
GIST
- Add more GIST index support for geometric data types
- Allow GIST indexes to create more complex index types, like digital trees (see Aoki)
- Fix performance issues in contrib/seg and contrib/cube GiST support
Hash
- Add UNIQUE capability to hash indexes
- Allow multi-column hash indexes
- This requires all columns to be specified for a query to use the index.
- Write Ahead Logging for Hash Indexes
Sorting
- Consider detoasting keys before sorting
- Allow sorts of skinny tuples to use even more available memory.
- Now that it is not limited by MaxAllocSize, don't limit by INT_MAX either.
Cache Usage
- Consider automatic caching of statements at various levels:
- Parsed query tree
- Query execute plan
- Query results
- Cached Query Plans (was: global prepared statements)
- PoC plpgsql - possibility to force custom or generic plan
- Cached/global query plans, autopreparation
- Consider allowing higher priority queries to have referenced shared buffer pages stay in memory longer
- Fix memory leak caused by negative catcache entries
- Better regexp cache management:
- This would be fairly isolated, but involve memory management, error handling, data structures, GUCs.
- Each backend has a cache of compiled regular expressions. If you run
SELECT 'abc' ~ 'a.c'
, thenSELECT ident FROM pg_backend_memory_contexts WHERE name = 'RegexpMemoryContext'
will show the cached expression. - Currently the size of the cache is fixed at MAX_CACHED_RES (32), and the array of cached expressions is searched linearly by RE_compile_and_cache(), with a performance mitigation that recently used expressions are moved to the front of the array. An application making heavy use of regular expressions might want to increase the limit, possibly expressing it using something like
SET regexp_cache_size='1MB'
, and to index it with a hash table, possibly using simplehash.h, while still maintaining an LRU list, possibly using dlist.
Vacuum
- Allow VACUUM FULL and CLUSTER to update the visibility map
- Improve tracking of total relation tuple counts now that vacuum doesn't always scan the whole heap
- Bias FSM towards returning free space near the beginning of the heap file, in hopes that empty pages at the end can be truncated by VACUUM
- Add a way to compact tables without exclusive locking, similar to pre-9.0 VACUUM FULL
- Provide more information in order to improve user-side estimates of dead space bloat in relations
- Reduce the number of table scans performed by vacuum
- Vacuum Gin indexes in physically order rather than logical order
- Avoid creation of the free space map for small tables
Auto-vacuum
- Issue log message to suggest VACUUM FULL if a table is nearly empty?
- Prevent long-lived temporary tables from causing frozen-xid advancement starvation
- The problem is that autovacuum cannot vacuum them to set frozen xids; only the session that created them can.
- Prevent autovacuum from running if an old transaction is still running from the last vacuum
- Have autoanalyze of parent tables occur when child tables are modified
- Allow visibility map all-visible bits to be set even when an auto-ANALYZE is running
- Improve autoanalyze thresholds for small tables
Locking
- Fix problem when multiple subtransactions of the same outer transaction hold different types of locks, and one subtransaction aborts
- Improve deadlock detection when a page cleaning lock conflicts with a shared buffer that is pinned
- Detect deadlocks involving LockBufferForCleanup()
- Allow finer control over who is cancelled in a deadlock
Startup Time Improvements
- Allow backends to change their database without restart
- This allows for faster server startup.
Write-Ahead Log
- Eliminate need to write full pages to WAL before page modification
-
Currently, to protect against partial disk page writes, we write full page images to WAL before they are modified so we can correct any partial page writes during recovery. These pages can also be eliminated from point-in-time archive files.
- Re: Index Scans become Seq Scans after VACUUM ANALYSE
- http://archives.postgresql.org/pgsql-hackers/2011-05/msg01191.php
- WIP double writes
- double writes
- Double-write with Fast Checksums
- double writes using "double-write buffer" approach
- http://archives.postgresql.org/pgsql-hackers/2012-10/msg01463.php
- When full page writes are off, write CRC to WAL and check file system blocks on recovery
- If CRC check fails during recovery, remember the page in case a later CRC for that page properly matches. The difficulty is that hint bits are not WAL logged, meaning a valid page might not match the earlier CRC.
- Write full pages during file system write and not when the page is modified in the buffer cache
- This allows most full page writes to happen in the background writer. It might cause problems for applying WAL on recovery into a partially-written page, but later the full page will be replaced from WAL.
- Allow WAL information to recover corrupted pg_controldata
- Speed WAL recovery by allowing more than one page to be prefetched
- This should be done utilizing the same infrastructure used for prefetching in general to avoid introducing complex error-prone code in WAL replay.
- Improve WAL concurrency by increasing lock granularity
- Have resource managers report the duration of their status changes
- Close deleted WAL files held open in *nix by long-lived read-only backends
- Use less bytes to store the information of WAL records.
-
The WAL record format is quite verbose, and often contains more bytes than necessary for redo and decoding. By updating the WAL format we can reduce the average WAL record size of e.g. pgbench by several percent.
- Updating the WAL infrastructure tracks progress on some active development
- Re: RFC: WAL infrastructure issues, updates and improvements
Optimizer / Executor
- Improve selectivity functions for geometric operators
- Consider adding a useful and convenient way of seeing paths rejected by the planner. OPTIMIZER_DEBUG does this so some extent but it seemingly gets very little use and must be enabled during compilation.
- Log statements where the optimizer row estimates were dramatically different from the number of rows actually found?
- Consider compressed annealing to search for query plans
- This might replace GEQO.
- Add planner support for cardinality-reducing functions
- estimate_num_groups currently assumes (under item 2 in its header comment) that any function on an attribute won't meaningfully decrease the cardinality (number of groups) generated from that attribute, i.e. num_groups(a) ~= num_groups(f(a)). However, for truncating functions such as int4mod, date_trunc, or date_bin, this assumption is wrong and will result in plans that can significnatly overestimate the number of groups.
Hashing
- Allow single batch hash joins to preserve outer pathkeys
- Avoid building the same hash table more than once during the same query
- Avoid hashing for distinct and then re-hashing for hash join
Background Writer
- Consider having the background writer update the transaction status hint bits before writing out the page
- Implementing this requires the background writer to have access to system catalogs and the transaction status log.
- Consider adding buffers the background writer finds reusable to the free list
- Automatically tune bgwriter_delay based on activity rather then using a fixed interval
- Consider whether increasing BM_MAX_USAGE_COUNT improves performance
- Test to see if calling PreallocXlogFiles() from the background writer will help with WAL segment creation latency
Concurrent Use of Resources
- Do async I/O for faster random read-ahead of data
- Async I/O allows multiple I/O requests to be sent to the disk with results coming back asynchronously. The above patch is already applied as of 8.4, but it still remains to figure out how to handle plain indexscans effectively.
TOAST
- Allow user configuration of TOAST thresholds
- Reduce unnecessary cases of deTOASTing
- Reduce costs of repeat de-TOASTing of values
Monitoring
- Have pg_stat_activity display query strings in the correct client encoding
- Allow reporting of stalls due to wal_buffer wrap-around
- Restructure pg_stat_database columns tup_returned and tup_fetched to return meaningful values
- Improve handling of pg_stat_statements handling of bind "IN" variables
Miscellaneous Performance
- Allow configuration of backend priorities via the operating system
- Though backend priorities make priority inversion during lock waits possible, research shows that this is not a huge problem.
- Consider if CommandCounterIncrement() can avoid its AcceptInvalidationMessages() call
- Consider Cartesian joins when both relations are needed to form an indexscan qualification for a third relation
- Consider not storing a NULL bitmap on disk if all the NULLs are trailing
- Sort large UPDATE/DELETEs so it is done in heap order
- Add auto-tuning of work_mem
- Consider decreasing the I/O caused by updating tuple hint bits
- Hint Bits and Write I/O
- Re: [HACKERS] Hint Bits and Write I/O
- http://archives.postgresql.org/pgsql-hackers/2010-10/msg00695.php
- http://archives.postgresql.org/pgsql-hackers/2010-11/msg00792.php
- http://archives.postgresql.org/pgsql-hackers/2011-01/msg01063.php
- http://archives.postgresql.org/pgsql-hackers/2011-03/msg01408.php
- http://archives.postgresql.org/pgsql-hackers/2011-03/msg01453.php
- Avoid reading in b-tree pages when replaying vacuum records in hot standby mode
- Restructure truncation logic to be more resistant to failure
- This also involves not writing dirty buffers for a truncated or dropped relation
- Enhance foreign data wrappers, parallelism, partitioning, and perhaps add a global snapshot/transaction manager to allow creation of a proof-of-concept built-in sharding solution
- Ideally these enhancements and new facilities will be available to external sharding solutions as well.
- Consider a built-in connection pooler
Miscellaneous Other
- Deal with encoding issues for filenames in the server filesystem
- Deal with encoding issues in the output of localeconv()
- Provide schema name and other fields available from SQL GET DIAGNOSTICS in error reports
- Use sa_mask to close race conditions between signal handlers
- Allow pg_export_snapshot() to run on hot standby servers
- This will allow parallel pg_dump on such servers.
- Provide a way to enumerate and unregister background workers
- Right now the only way to unregister bgworkers is from within the worker with proc_exit(0) or registering with BGW_NEVER_RESTART
Source Code
- Rationalize division of labor between initdb and bootstrap
- Allow cross-compiling by generating the zic database on the target system
- Improve NLS maintenance of libpgport messages linked onto applications
- Use UTF8 encoding for NLS messages so all server encodings can read them properly
- Consider making NAMEDATALEN more configurable
-
There is demand for making 128 the default, but there are also concerns about storage and memory usage and performance. So a rearchitecting to make the storage variable-length might be preferred.
- Discussions when it was changed from 32 to 64: [4] [5] [6] [7]
- Revisiting NAMEDATALEN
- NAMEDATALEN increase because of non-latin languages (contains ideas about variable-length storage)
- Research use of signals and sleep wake ups
- Consider simplifying how memory context resets handle child contexts
- Implement the non-threaded Avahi service discovery protocol
- Reduce data row alignment requirements on some 64-bit systems
- Restructure TOAST internal storage format for greater flexibility
- Consider removing the attribute options cache
- Restructure /contrib section
Windows
- Allow psql to use readline once non-US code pages work with backslashes
- Improve signal handling
- Support PGXS when using MSVC
- Fix MSVC NLS support, like for to_char()
- Fix global namespace issues when using multiple terminal server sessions
- Improve consistency of path separator usage
- Fix cross-compiling on Windows
- Reduce file statistics overhead on directory reads
- Fix hang with long file paths
Wire Protocol Changes / v4 Protocol
- Allow dynamic character set handling
- Ensure the client can determine the encoding of messages sent early in the handshake
- Let the client indicate character encoding of database names, user names, passwords, and of pre-auth error messages returned by the server
- Send numeric version to clients in fixed header
- Add decoded type length/precision (i.e. typmod information)
- Mark result columns as known-not-null when possible
- Use compression
- Specify and implement wire protocol compression. If SSL transparent compression is used, hopefully avoid the overhead of key negotiation and encryption when SSL is configured only for compression. Note that compression is being removed from TLS 1.3 so we really need to do it ourselves.
- Update clients to use data types, typmod, schema.table.column names of result sets using new statement protocol
- Set protocol for wire format negotiation
- Make sure upgrading to a 4.1 protocol version will actually work smoothly
- Allow multi-state authentication (e.g. try client peer, fall back to md5)
- Allow re-authentication
- Let the client request re-authentication as a different user mid session, for connection pools that pass through the handshake.
- Identify the affected object in CommandComplete message?
- Allow negotiation of encryption, STARTTLS style, rather than forcing client to decide on SSL or !SSL before connecting
- Permit lazy fetches of large values, at least out-of-line TOASTED values
- Add session-level whitelisting of types for binary-mode transfer
- Send client the xid when it is allocated
- Lets the client later ask the server "did this commit or not?" after interterminate result due to crash or connection loss
- Report xlog position in commit message
- Help enable client-side failover by providing a token clients can use to see if a commit has replayed to replicas yet
- Changes to make cancellations more reliable and more secure
- Clarify semantics of statement_timeout in extended query protocol
- Batched and pipelined queries have unexpected behaviour with statement_timeout. Client needs to be able to specify statement boundary with protocol message.
- Create a more efficient way to handle out-of-line parameters
- Separate transaction delineation from protocol error recovery (in v3 both are managed via the same Sync message)
Documentation
- Provide a manpage for postgresql.conf
- Document support for N' ' national character string literals, if it matches the SQL standard
Exotic Features
- Add pre-parsing phase that converts non-ISO syntax to supported syntax
- This could allow SQL written for other databases to run without modification.
- Add features of Oracle-style packages
- A package would be a schema with session-local variables, public/private functions, and initialization functions. It is also possible to implement these capabilities in any schema and not use a separate "packages" syntax at all.
- Add autonomous transactions
- Give query progress indication
