PGXN v2
PGXN v2 is the code name for an over-arching project to re-think and re-implement PGXN.
Background
PGXN was created in 2011 by Theory, who designed and implemented it after CPAN, but for Postgres extensions. After an initial flurry of activity and releases in early 2012, the service has remained relatively stable and unchanged, chugging along, distributing extensions and periodic bug fixes and maintenance as time allowed.
PGXN has been a moderate success, at best. It has reliably served its purpose for 12 years, but never fulfilled its goal to be the source of record for open source Postgres extensions. As of Spring 2024, its About Page counts 390 extensions, while a manually-maintained list of extensions counts over 1000 publicly-available extensions. Some popular extensions, such as PostGIS, have never been published on PGXN.
Moreover, extensions have been difficult to install, especially for newcomers. The Yum and Apt repositories provide a few hundred easy-to-install extensions, but it can be difficult to find them, since they're not listed on PGXN or other registries on the web.
A number of new entrants arrived in 2023, including trunk, dbdev, and (in early 2024), pgxman. These extension registries focus less on distribution than on ease of use: while PGXN has been great for source code distribution, these services aimed to greatly simplify installation by building and providing downloadable binaries for select OSes and architectures. Where the pgxn client requires a complete compiler toolchain and Postgres devel features to build a PGXS extension, these registries' clients just download and install binary packages (or, for dbdev, TLEs) directly.
This has helped improve the lead time from discovery to installation to experimentation for a limited number of extensions on a short list of platforms. But clearly there's appetite for a canonical, searchable index of all extensions with support for binary installation.
A Rethinking
In January 2024, Theory started work at Tembo with the charge to rethink the role and architecture of PGXN in the face of increased need for an authoritative source and binary distribution. The immediate result is this project, code-named "PGXN v2".
This page will be periodically updated with links and updates as design, planning, and implementation progress. That work takes place across a variety of blog posts, GitHub repositories, Gists, mail lists, Slacks, Discourses, and conference sessions. Here we'll make an effort to link to them all, to serve as a sort of central index and point of departure for discussion and feedback.
Status
We're currently seeking reviews, feedback, questions, and commentary on PGXN v2 Architecture, a document that defines the services to be provided and the architecture to run them, as well as the strategic vision to guide project planning and decision-making.
From the introduction:
This document outlines the project to build extension distribution, discovery, and packaging tools and services to power the growth, accessability, and utility of the Postgres extension ecosystem. Taking the overall Postgres community as its audience, it defines the services to be provided and the architecture to run them, as well as the strategic vision to guide project planning and decision-making.
With the goal to think strategically and plan pragmatically, this document describes the former to enable the latter. As such, it is necessarily high-level; details, scoping, and planning will be surfaced in more project-focused documents.
Bear in mind that this document outlines an ambitious, long-term strategy. If you're thinking that there's too much here, that we'er over-thinking and over-designing the system, rest assured that project execution will be fundamentally incremental and pragmatic. This document is the guiding light for the project, and subject to change as development proceeds and new wrinkles arise.
Index
RFCS
- Binary Distribution Format
- This RFC specifies the binary distribution format for PGXN packages, also called the trunk format. Inspired by the Python wheel format and Trunk.
- Meta Spec v2
- This RFC significantly updates the existing v1 spec to better identify different types of extensions, build binary packages, and manage dependencies.
Design Documents
- Extension Ecosystem: Jobs and Tools
- The challenges of the current Postgres extension ecosystem and the interest and energy put into exploring new solutions make clear that the time has come to revisit the whole idea. We begin with a survey of the jobs to be done by extensions packaging and distribution, followed by an enumeration of the tools and services anticipated to make it a reality.
- PGXN v2 Architecture
- Outlines the project to build extension distribution, discovery, and packaging tools and services to power the growth, accessability, and utility of the Postgres extension ecosystem. Taking the overall Postgres community as its audience, it defines the services to be provided and the architecture to run them, as well as the strategic vision to guide project planning and decision-making.
- Service Disposition
- A summary of the ambitiously-envisioned future PGXN services and architecture, followed by an examination of existing services and how they will gradually be refactored or replaced for the updated platform.
Blog Posts
In revers chronological order.
- Release: pgxn_meta v0.1.0
- Announcing the release of pgxn_meta v0.1.0, a Rust crate and executable for validating PGXN Meta v1 and v2 META.json files.
- To Preload, or Not to Preload
- When should a Postgres extension be pre-loaded and when should it not? Should it be loaded in user sessions or at server startup?
- POC: Distributing Trunk Binaries via OCI
- Would it be possible to distribute Postgres extension binaries via Open Container Registries? Tune in to find out!
- POC: PGXN Binary Distribution Format
- A proof of concept for “trunk”, the proposed binary distribution format for Postgres extensions defined in RFC 2.
- Álvaro Hernández: Why Postgres Extensions should be packaged and distributed as OCI images
- It’s all about not reinventing the wheel, and leveraging the ecosystem around OCI. Many of the problems (solutions) in building, packaging and distributing extensions are already solved by OCI: there’s a whole ecosystem of tools around OCI that provide additional benefits in terms of tooling, infrastructure and common knowledge.
- PGXN v2: Go or Rust?
- What programming language(s) should we use to build new and revamp existing PGXN services and tools: Rust or Go? [Poll closed.]
- RFC: PGXN Metadata Sketch
- Request for comments on a sketch of a new metadata standard for Postgres extension packaging, distribution, and delivery, building on the PGXN Meta Spec to address its shortcomings and emerging use cases 12 years on.
- Extension Registry Namespacing RFC
- A proposal for an additional level of name uniqueness for Postgres extension packaging and distribution, based on URIs.
- Major Developments in Postgres Extension Discovery and Distribution
- We want to hear feedback and thoughts from everyone, large and small — to make the extension experience better for both users and developers alike.
- RFC: Extension Metadata Typology
- Thinking through the PostgreSQL extension metadata use cases and recognizing the types of information they need.
- The History and Future of Extension Versioning
- What versioning standard should be used for Postgres extension distribution? Some context from PostgreSQL and PGXN, a survey of the version standard landscape today, and a recommendation.
- Extension Ecosystem Jobs to be Done
- The challenges of the current Postgres extension ecosystem and the interest and energy put into exploring new solutions make clear that the time has come to revisit the whole idea. We begin with a survey of the jobs to be done by extensions packaging and distribution.
- Contemplating Decentralized Extension Publishing
- The Go package ecosystem uses distributed publishing to release modules without authentication or uploads. Could we do something similar for Postgres extensions?
- PGXN Challenges
- Some thoughts on the challenges for PGXN’s role in the ideal PostgreSQL extension ecosystem of the future.
Presentations
- State of the Postgres Extension Ecosystem
- Video of a presentation Theory made at POSETTE
- The future of the extension ecosystem
- Video of a presentation Theory made at PGConf.dev.
- Presentation: Introduction to the PGXN Architecture (2024-02-01)
- Video of a presentation Theory made to the Tembo team on the PGXN architecture.
Ecosystem Summit
The Extension Ecosystem Summit 2024 was an in-person event at PGConf.dev 2024. See report for a summer of topics addressed, and the Summit notes for all the details.
The Summit was preceded by a series of virtual mini-talks and community discussions, referred to as mini summits.
- David Wheeler on the State of the Extension Ecosystem (2024-03-06)
- Video, slides, and notes from the first Extension Ecosystem Mini-Summit, covering the history history and future of PostgreSQL extensability and PGXN.
- Ian Stanton on Building Trunk (2024-03-20)
- Video, slides, and transcript from the second Extension Ecosystem Mini-Summit, coevering the motivations, design, architecture, and lessons learned building the trunk binary extension registry.
- Devrim Gündüz on RPMifying Extensions (2024-04-03)
- Video, slides, and transcript from the third Extension Ecosystem Mini-Summit, coevering the history, design, and maintenance of the community Yum repository and the implications for distributing extensions via Yum, Apt, and more.
- Jonathan Katz on Trusted Language Extensions (2024-04-07)
- Video, slides, and transcript from the fourth Extension Ecosystem Mini-Summit, covering the vision and specifics behind Trusted Language Extensions, a.k.a. [TLE https://github.com/aws/pg_tle]s.
- Yurii Rashkovskii on Universally Buildable Extensions (2024-05-01)
- Video, slides, and transcript from the fifth Extension Ecosystem Mini-Summit, in which Yurii Rashkovskii describes a POC that discovers how to build extensions, as well as potential challenges when the Postgres C source changes in minor releases.
- Community Organizing Summit Topics (2024-04-17)
- Video, slides and a rough transcript of the sixth and final Extension Ecosystem Mini-Summit, in which we review potential topics for the in-person summit at PGConf.dev and discuss how to organize it.
