SkyTools

From PostgreSQL wiki

Revision as of 21:27, 18 May 2012 by Boshomi (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

This is a package of tools in use in Skype for replication and failover. It includes a generic queuing mechanism, PgQ and utility library for Python scripts.

The source code and documentation is hosted at the pgfoundry project page.

A general overview presentation can be found here: File:Moskva DB Tools.v3.pdf, and a PGQ detailed presentation is at http://www.pgcon.org/2008/schedule/events/79.en.html.

Contents

PgQ

This is the queue machanism we use. It consists of PL/pgsql, and C code in database, with Python framework on top of it. It is derived from snapshot-based event handling ideas from Slony-I, re-written for more general use.

Features

  • A database can have many queues.
  • Producers can insert into any queue.
  • There can be several consumers for any given queue and all consumers see all events.

Documentation

Monitoring

A PGQ Munin Plugin is available.

See Also

PGQ can be the base solution to implement a BASE solution (basically available, soft state, eventually consistent), or more generally to implement an Eventually Consistent system.

Kristo's blog host a detailed example of a python PGQ consumer which sends confirmation emails after user inscription: http://kaiv.wordpress.com/2007/10/19/skytools-database-scripting-framework-pgq/

Londiste

Replication tool written in Python, using PgQ as event transport.

Features

  • Tables can be added one-by-one into the replication set.
  • The initial COPY for one table does not block event replay for other tables.
  • Can compare tables on both sides.

Documentation

Monitoring

Londiste is "just" a PGQ Consumer, so see about how to monitor the queue itself, with the PGQ_Munin_Plugin e.g.

walmgr

This script sets up WAL archiving, does initial backup and runtime WAL archive and restore.

See its documentation.

Provided specific consumers

Some more tools based on Skytools Framework are available for handling specific jobs :

Each documentation page begins with a description section, but to ease your browsing experience, they are all in the same page here: Skytools Specific Consumers.

Personal tools