PgConf.Russia 2017 talks

Test environment on demand

Radoslav Glinsky
Skype (Microsoft), Software developer - PostgreSQL tooling

Since 2011 I've been working on databases at Skype. I started as PostgreSQL developer/tester on projects for Skype contacts, authentication and integrations. Since 2013 I am in the internal Database Platform team, which provides "PostgreSQL as service" for the development teams. My job includes automation and tooling for DB deployments, replication, asynchronous DB scripting, as well as development of an internal DB warehouse.

Do you test your PostgreSQL releases prior to Production in a dedicated test environment? Are you sure that your test environment (shortly Test) is equal to Production and in an appropriate state?

In Skype we were facing multiple challenges associated with database testing:
- Simplifying complex Production architecture of thousands of PostgreSQL instances, interconnected with RPCs and replications, infrastructure servers and external DB scripts, into their Test counterparts.
- Constantly growing hardware requirements, insufficient cleanup of data generated in Test.
- Differences between Test and Production were appearing and accumulating. Recognizing and fixing them required lots of effort.
Instead of fighting the accumulating issues, we have come with a new idea in Skype - to rebuild our PostgreSQL Test from scratch, based on Production! And we do it regularly, or whenever on demand. In fully automated way we clone Production into a lightweight Test - restoring data structures, procedures, roles, replications, external database scripts, configurations, etc., even the failover databases. For individual test-specific data and settings, we've developed a way how to retain those over rebuilds.

As the result, nowadays we have confidence that our Test is equal to Production. It doesn't go stale or get bloated. And we can easily spawn as many copies of test environments as needed. Maintaining Test is much easier for us nowadays and we want to share our experience with further PostgreSQL community.

VIDEO

Slides