HowToBetaTest

From PostgreSQL wiki
Jump to navigationJump to search


How to Beta Test

If you are not a code contributor (or even if you are) testing PostgreSQL Betas is one of the best things you can do for our project. By participating in organized testing, you help get the release out faster, with more features and fewer bugs.

Prerequisites to Testing

A. Have a Test Machine: somewhere you can run tests on a PostgreSQL database and/or application, even if they may result in using all of your computer's system resources, or crashing. If your company uses PostgreSQL and you have a testbed for development changes, this is ideal.

B. Have a Test: an application test, interface test, performance test, and/or database feature test you can run. See below for explanations of this or suggestions.

C. Have Some Time: for testing when new Alphas or Betas come out. It doesn't have to be a lot of time, but do expect to spend a couple hours testing, and following up on questions from PostgreSQL hackers, with each release.

Particular Things to Test in 9.6

9.6 has many new features and many changes. Here's a few things which the PostgreSQL developers think might need special attention.

New Features to Test

The following are brand new features. Please try them out; do they function as documented? Do they work with each other? Are the defaults and parameters reasonable (if applicable)? Can the new features be use with each other?

  • Parallel Query
    • Parallel sequential scans
    • parallel joins
    • parallel aggregates
  • postgres_fdw
    • Sort pushdown
    • join pushdown
    • DML (UPDATE/DELETE) pushdown
    • Extension cascade support to install dependencies
  • Synchronous Replication
    • New remote_apply replication mode which waits for confirmation that a standby has applied changes
    • Support for multiple synchronous standbys
  • Transactions, VACUUM and the Visibility Map
    • pg_visibility extension for examining visibility maps
    • Frozen page data in visibility map for skipping vacuum on already-frozen data
    • User-defined expiration of snapshots to control table bloat
  • Other Features
    • Cube extension kNN support
    • Command progress reporting
    • Detailed wait information in pg_stat_activity
    • Index-only scans for partial indexes
    • Phrase full text search

See the What's New wiki page for some information about what to do with these features, as well as the Release Notes

Performance Improvements

The following areas of PostgreSQL performance are supposed to be significantly improved. Are they improved for you? Is performance actually worse? Please test.

  • Large speed improvements in sorting text and numeric fields for CREATE INDEX, CLUSTER, and query execution. Includes internal and external sorts.
    • The "abbreviated keys" optimization can provide useful debugging information, by running "set trace_sort = on;" and consulting the server log (or by running "client_min_messages = 'LOG'" from psql to see log output directly from the psql client). The per-type instrumentation can give a good indication of why the optimization might not work out for some particular dataset.
  • Improvements to hash creation and lookup performance
  • Large increases in multi-core and large memory scalability
  • WAL compression for smaller/faster transaction logs

Possible Regressions

The following areas have been flagged as areas which could result in regressions or crashes on some users' platforms and applications. Please test if they are regressions for you:

  • Sorting: do the sorting improvements fail or regress on your platform/encoding?
  • Test failover and recovery from binary backup (PITR) with WAL compression turned on. Does it still work? Is there data corruption?
  • Do the changes in operator precedence change the results of your queries?
  • Does the change to casting behavior in PL/pgSQL break your functions?
  • Do REASSIGN OWNED and ALTER OWNER TO now set permissions correctly on changed objects?
  • Can you break into Postgres by exploiting a new security hole? Please report all security issues to pgsql-security@postgresql.org.
  • Does backup and restore work correctly with an RLS-secured database?

Documentation Review

We strive to document new features as best as possible, but errors may slip in. Also, documentation might be lacking some details, or be unclear or misleading. You can review newly-added documentation with a command like:

  • git log -p remotes/origin/REL_11_STABLE..master -- doc ; or,
    • This shows individual commits, which includes intermediate changes and bugs (like: implement feature ; rearrange docs ; fix typo ; ...).
  • git diff remotes/origin/REL_11_STABLE..master -- doc
    • This shows only the difference between docs from previous version and version to be tested, so loses commit granularity, but avoids showing intermediate issues already resolved by another commit.

Known issues

A number of known issues made it into PostgreSQL 9.6 Beta. These open issues are separately documented. They may provide some inspiration when seeking out areas of the code to find similar bugs in.

Types of Tests You Can Run

Install Test

This is the simplest one, and precedes all of the others. Install the new version of PostgreSQL. Configure it with your favorite options and contrib modules and external tools. Does it all work?

Custom Application Tests

If you're already using PostgreSQL in production on your custom application, try porting your application to the new release and run your own application tests and performance tests. Try upgrading a copy of a populated database and seeing if there are any upgrade errors. Tell us what you find.

This is more valuable, of course, if your application has a full testing framework which allows you to unit test your application functions and its overall performance. But even doing some ad-hoc user testing of the upgrade is helpful.

General Application Tests

Some popular open source and proprietary applications come with their own test suites, unit tests, and/or performance tests. At some point, we will have a list of these; if you know one, please add it. For each of these, you can test the following things:

  • Upgrading a populated database
  • Create a new database using the software's installer
  • Run unit tests and look for errors
  • Run performance tests and look for performance regression

Open Source applications with test frameworks:

  • Rails Apps:
  • Python Applications:
    • Django
  • ORMs
    • SQL Alchemy
    • DataObjects
    • DataMapper
    • ActiveRecord
  • PHP Apps:
    • phpPgAdmin
  • Perl Apps:

Interface Tests

Nobody can use PostgreSQL without a driver, and we want to make sure that all of our drivers work as well as possible with each new version of PostgreSQL. We particularly want to make sure that new features don't break these drivers. Fortunately, many drivers have regression tests.

So, download the latest version of your favorite driver (and maybe some others you don't like as much, but are still current/popular) point them at the new release of PostgreSQL, and run the regression tests. Then, build a simple database with some of the new release's features, and try to query them; do new syntax or new data types break the driver?

Performance Tests

Another critical thing to check for is performance regressions and improvements. The difficulty about testing for this is that you really need to be sure to conduct your tests in a manner which produces repeatable results, or the performance metrics aren't worthwhile. This means that you should run them on a system which has nothing else running other than the test, you should run the test several times and see if results are consistent, etc. You also need to run the test or benchmark both against the previous version (e.g. 8.4.4) and the current alpha or beta.

Performance tests you can run include:

  • An application performance benchmark, if your custom or public application has one.
  • pgbench. Instructions on getting useful results from pgbench up later.
  • Real benchmarks like DBT2, Spec's EAStress, etc. These require a large investment in time and server hardware to run.
  • Specific task peformance; try running an operation which you know strains PG's resources (like a 40-table JOIN or an ELT operation) or one which is supposed to be improved in the new release.

If you start running performance tests, it's very important that you stick with it for the whole alpha/beta cycle so that we can see if we're improving and/or fixing any problems.

Feature Tests

Every alpha or beta release has new features. While our contributors do a good job of testing each feature in isolation, the combinatorics of testing each new feature with every other new feature, as well as with old features, is impossible for the hackers. Also, we're so close to the code that we can miss leaving major things out of the docs. This is where you come in. Things to test:

  • Pick any new user-facing feature you've never heard of before. Try to use it based entirely on the docs. Report results, problems, and suggested doc changes.
  • Pick any two or three new features which could be combined somehow ( e.g. join removal and partition joins, or DefaultACLs and column triggers). Try to run a script or query which combines these features. Look for errors or non-spec or confusing behavior. Report both the testing procedure and the results.
  • Pick any new feature and an old feature which was either complex (e.g. Tsearch2) or added in the last few versions (e.g. Windowing Functions). Combine them, report how you did it and the results.

Again, it's important here that you follow up on fixes for any issues you report.

How to Test

1. The first time you're going to run a particular test, you might want to post it to pgsql-testers so that other testers can suggest tweaks to make the test more useful.

2. Script the test so that you can reproduce it.

3. Run the test and collect the results to a log.

3.a. Run the test on an older version for comparison, if applicable.

4. Copy the e-mail template below and post the results using the template to pgsql-testers.

5. Look for follow-up on your test and respond to it.

6. Wait for the next release and run the test again.

Reporting Tests

You can report tests by email. You can subscribe to any PostgreSQL mailing list from the subscription form. Note you can also post to mailing lists without subscribing.

  • pgsql-bugs: this is the preferred mailing list if you think you have found a bug in the beta. You can also use the Bug Reporting Form.
  • pgsql-general: most users who are doing testing should report their testing here.
  • pgsql-hackers: bugs, questions, and successful test reports are welcome here if you are already subscribed to pgsql-hackers. Note that pgsql-hackers is a high-traffic mailing list with a lot of development discussion.

What to report

Please try to include all of the following information in your email:

Name:

Release:


Test Type:


Test Detail:


Platform:


Installation Method:


Platform Detail:


Test Procedure:


Failure?


Test Results:


Comments:


What goes in the fields:

Name Your name, so that we can track who is doing which tests, so if you use a nickname, please be consistent. For that matter, if you do a lot of beta-testing you may be credited in the release notes and/or the contributors page, so you may want to give us your real name.

Release: the specific release you tested. Required.

Test Type: What kind of test was it? Mark multiple options if it was a combination test.

Test Detail: A short description of the test you ran. e.g., "SQLAlchemy Unit Test Suite" or "pgBench" or "Combined sync replication and unlogged tables". Required.

Platform: Your operating system.

Installation Method: how did you install the alpha or beta release?

Platform Detail: A brief description of the platform you ran the test on. e.g. "MacBook Snow Leopard 64-bit" or "Linux 16-core Sun 4600 with NAS storage". Optional.

Test Procedure: steps to reproduce your test, if it's not immediately obvious. Optional.

Failure: Did the test show a compatibility issue, an error, or a performance regression? Or anything else the PostgreSQL hackers should take a look at? Required.

Results: A narrative of how you ran the test and what happened. If you experienced errors, please paste detailed error messages.

Comments: Anything else relevant which wasn't included above. Optional.

Retrieved from "https://wiki.postgresql.org/index.php?title=HowToBetaTest&oldid=33188"