SkyTools
This is a package of tools in use in Skype for replication and failover. It includes a generic queuing mechanism, PgQ and utility library for Python scripts.
The source code was hosted on Pgfoundry (archived) and later on GitHub, tech lead is Marko Kreen.
A general overview presentation can be found here: File:Moskva DB Tools.v3.pdf, and a PGQ detailed presentation is at http://www.pgcon.org/2008/schedule/events/79.en.html.
PgQ
This is the queue mechanism we use. It consists of PL/pgsql, and C code in database, with Python framework on top of it. It is derived from snapshot-based event handling ideas from Slony-I, re-written for more general use.
Features
- A database can have many queues.
- Producers can insert into any queue.
- There can be several consumers for any given queue and all consumers see all events.
Documentation
Monitoring
A PGQ Munin Plugin is available.
See Also
PGQ can be the base solution to implement a BASE solution (basically available, soft state, eventually consistent), or more generally to implement an Eventually Consistent system.
Kristo's blog host a detailed example of a python PGQ consumer which sends confirmation emails after user inscription: http://kaiv.wordpress.com/2007/10/19/skytools-database-scripting-framework-pgq/
Londiste
Replication tool written in Python, using PgQ as event transport, available on Github. GitHub
Features
- Tables can be added one-by-one into the replication set.
- The initial COPY for one table does not block event replay for other tables.
- Can compare tables on both sides.
Documentation
- Londiste Tutorial
- Skytools docs are no longer available
Monitoring
Londiste is "just" a PGQ Consumer, so see about how to monitor the queue itself, with the PGQ_Munin_Plugin e.g.
walmgr
This script sets up WAL archiving, does initial backup and runtime WAL archive and restore.
See its documentation.
Provided specific consumers
Some more tools based on Skytools Framework are available for handling specific jobs :
Each documentation page begins with a description section, but to ease your browsing experience, they are all in the same page here: Skytools Specific Consumers.
