PGXN v2
PGXN v2 is the code name for an over-arching project to re-think and re-implement PGXN.
Background
PGXN was created in 2011 by Theory, who designed and implemented it after CPAN, but for Postgres extensions. After an initial flurry of activity and releases in early 2012, the service has remained relatively stable and unchanged, chugging along, distributing extensions and periodic bug fixes and maintenance as time allowed.
PGXN has been a moderate success, at best. It has reliably served its purpose for 12 years, but never fulfilled its goal to be the source of record for open source Postgres extensions. As of Spring 2024, its About Page counts 390 extensions, while a manually-maintained list of extensions counts over 1000 publicly-available extensions. Some popular extensions, such as PostGIS and pgvector have never been published on PGXN.
Moreover, extensions have been difficult to instal especially for newcomers. The Yum and Apt repositories provide a few hundred easy-to-install extensions, but it can be difficult to find them, since they're not listed on PGXN or other registries on the web.
A number of new entrants arrived in 2023, including trunk, dbdev, and (in early 2024), pgxman. These extension registries focused less on distribution than on ease of use: while PGXN has been great for source code distribution, these services aimed to greatly simplify installation by building and providing downloadable binaries for select OSes and architectures. Where the pgxn client requires a complete compiler toolchain and Postgres devel features to build a PGXS extension, these registries clients just download and install binary packages (or, for dbdev, TLEs) directly.
This has helped improve the lead time from discovery to installation to experimentation for a limited number of extensions on a short list of platforms. But clearly there's appetite for a canonical, searchable index of all extensions with support for binary installation.
A Rethinking
In January 2024, Theory started work at Tembo with the charge to rethink the role and architecture of PGXN in the face of increased need for an authoritative source and binary distribution. The immediate result is this project, code-named "PGXN v2".
This page will be periodically updated with links and updates as design, planning, and implementation progress. That work takes place across a variety of blog posts, GitHub repositories, Gists, mail lists, Slacks, Discourses, and conference sessions. Here we'll make an effort to link to them all, to serve as a sort of central index and point of departure for discussion and feedback.
Status
We're currently seeking reviews, feedback, questions, and commentary on PGXN v2 Architecture, a document that defines the services to be provided and the architecture to run them, as well as the strategic vision to guide project planning and decision-making.
From the introduction:
This document outlines the project to build extension distribution, discovery, and packaging tools and services to power the growth, accessability, and utility of the Postgres extension ecosystem. Taking the overall Postgres community as its audience, it defines the services to be provided and the architecture to run them, as well as the strategic vision to guide project planning and decision-making.
With the goal to think strategically and plan pragmatically, this document describes the former to enable the latter. As such, it is necessarily high-level; details, scoping, and planning will be surfaced in more project-focused documents.
Bear in mind that this document outlines an ambitious, long-term strategy. If you're thinking that there's too much here, that we'er over-thinking and over-designing the system, rest assured that project execution will be fundamentally incremental and pragmatic. This document is the guiding light for the project, and subject to change as development proceeds and new wrinkles arise.
Index
Design Documents
- Extension Ecosystem: Jobs and Tools
- The challenges of the current Postgres extension ecosystem and the interest and energy put into exploring new solutions make clear that the time has come to revisit the whole idea. We begin with a survey of the jobs to be done by extensions packaging and distribution, followed by an enumeration of the tools and services anticipated to make it a reality.
- PGXN v2 Architecture
- Outlines the project to build extension distribution, discovery, and packaging tools and services to power the growth, accessability, and utility of the Postgres extension ecosystem. Taking the overall Postgres community as its audience, it defines the services to be provided and the architecture to run them, as well as the strategic vision to guide project planning and decision-making.
- Service Disposition
- A summary of the ambitiously-envisioned future PGXN services and architecture, followed by an examination of existing services and how they will gradually be refactored or replaced for the updated platform.
Blog Posts
- PGXN Challenges
- Some thoughts on the challenges for PGXN’s role in the ideal PostgreSQL extension ecosystem of the future.
- Contemplating Decentralized Extension Publishing
- The Go package ecosystem uses distributed publishing to release modules without authentication or uploads. Could we do something similar for Postgres extensions?
- Extension Ecosystem Jobs to be Done
- The challenges of the current Postgres extension ecosystem and the interest and energy put into exploring new solutions make clear that the time has come to revisit the whole idea. We begin with a survey of the jobs to be done by extensions packaging and distribution.
- The History and Future of Extension Versioning
- What versioning standard should be used for Postgres extension distribution? Some context from PostgreSQL and PGXN, a survey of the version standard landscape today, and a recommendation.
- RFC: Extension Metadata Typology
- Thinking through the PostgreSQL extension metadata use cases and recognizing the types of information they need.
- Major Developments in Postgres Extension Discovery and Distribution
- We want to hear feedback and thoughts from everyone, large and small — to make the extension experience better for both users and developers alike.
- Extension Registry Namespacing RFC
- A proposal for an additional level of name uniqueness for Postgres extension packaging and distribution, based on URIs.
- RFC: PGXN Metadata Sketch
- Request for comments on a sketch of a new metadata standard for Postgres extension packaging, distribution, and delivery, building on the PGXN Meta Spec to address its shortcomings and emerging use cases 12 years on.
- PGXN v2: Go or Rust?
- What programming language(s) should we use to build new and revamp existing PGXN services and tools: Rust or Go? [Poll closed.]
- Álvaro Hernández: Why Postgres Extensions should be packaged and distributed as OCI images
- It’s all about not reinventing the wheel, and leveraging the ecosystem around OCI. Many of the problems (solutions) in building, packaging and distributing extensions are already solved by OCI: there’s a whole ecosystem of tools around OCI that provide additional benefits in terms of tooling, infrastructure and common knowledge.
Presentations
- Presentation: Introduction to the PGXN Architecture (2024-02-01)
- Video of a presentation Theory made to the Tembo team on the PGXN architecture.
Ecosystem Summit
The Extension Ecosystem Summit 2024 is an in-person event at PGConf.dev in Vancouver on May 28. It is preceded by a series of vitual mini-talks and community discussions, referred to as mini summits.
- David Wheeler on the State of the Extension Ecosystem (2024-03-06)
- Video, slides, and notes from the first Extension Ecosystem Mini-Summit, covering the history history and future of PostgreSQL extensability and PGXN.
- Ian Stanton on Building Trunk (2024-03-20)
- Video, slides, and transcript from the second Extension Ecosystem Mini-Summit, coevering the motivations, design, architecture, and lessons learned building the trunk binary extension registry.
- Devrim Gündüz on RPMifying Extensions (2024-04-03)
- Video, slides, and transcript from the third Extension Ecosystem Mini-Summit, coevering the history, design, and maintenance of the community Yum repository and the implications for distributing extensions via Yum, Apt, and more.
- 2024-04-17: Jonathan Katz — “TLE Vision and Specifics”
- 2024-05-01: Yurii Rashkovskii — “Universally buildable extensions: dev to prod”
- 2024-04-17: TBD
