SkyTools

This is a package of tools in use in Skype for replication and failover. It includes a generic queuing mechanism, PgQ and utility library for Python scripts.

The source code was hosted on Pgfoundry (archived) and later on GitHub, tech lead is Marko Kreen.

A general overview presentation can be found here: File:Moskva DB Tools.v3.pdf, and a PGQ detailed presentation is at http://www.pgcon.org/2008/schedule/events/79.en.html.

PgQ

This is the queue mechanism we use. It consists of PL/pgsql, and C code in database, with Python framework on top of it. It is derived from snapshot-based event handling ideas from Slony-I, re-written for more general use.

Features

• A database can have many queues.
• Producers can insert into any queue.
• There can be several consumers for any given queue and all consumers see all events.

Documentation

• PGQ Tutorial
• PgQ admin tool (pgqadm)
• PgQ SQL API overview
• PgQ SQL reference

Monitoring

A PGQ Munin Plugin is available.

See Also

PGQ can be the base solution to implement a BASE solution (basically available, soft state, eventually consistent), or more generally to implement an Eventually Consistent system.

Kristo's blog host a detailed example of a python PGQ consumer which sends confirmation emails after user inscription: http://kaiv.wordpress.com/2007/10/19/skytools-database-scripting-framework-pgq/

Londiste

Replication tool written in Python, using PgQ as event transport, available on Github. GitHub

Features

• Tables can be added one-by-one into the replication set.
• The initial COPY for one table does not block event replay for other tables.
• Can compare tables on both sides.

Documentation

• Londiste Tutorial
• Skytools docs are no longer available

Monitoring

Londiste is "just" a PGQ Consumer, so see about how to monitor the queue itself, with the PGQ_Munin_Plugin e.g.

walmgr

This script sets up WAL archiving, does initial backup and runtime WAL archive and restore.

See its documentation.

Provided specific consumers

Some more tools based on Skytools Framework are available for handling specific jobs :

• Bulk Loader
• Cube Dispatcher
• Queue Mover
• Queue Splitter
• Table Dispatcher

Each documentation page begins with a description section, but to ease your browsing experience, they are all in the same page here: Skytools Specific Consumers.