PgConf.Russia 2017

The report is focused on the topic of automation of migration to PostgreSQL from other databases using Ispirer Migration and Modernization Toolkit. The issues of data and SQL code migration, conversion of client applications, embedded SQL and database API will be covered, the examples of implemented projects of migration from Oracle to PostgreSQL and from Microsoft SQL Server to PostgreSQL will be given.

VIDEO

Most of the main Skype services use a database platform based on PostgreSQL and other open-source technologies, such as Skytools, plProxy, pgBouncer, etc. This platform consists of several hundreds of servers with thousands of databases, which process hundreds of thousands of transactions per second. At the same time, the platform architecture allows its users (applications and their developers) to work with "logical" databases, without any worries about their real “physical” structure.

Our Skype Database Platform team is responsible for the database platform infrastructure. We develop automation systems for various processes that help us ensure service reliability and facilitate development, testing, and deployment of code. In this presentation, I will outline the database platform architecture, review its main components, and tell you about the methods we use in our every-day work to ensure high availability, scalability, replication, fault tolerance, and more.

Do you test your PostgreSQL releases prior to Production in a dedicated test environment? Are you sure that your test environment (shortly Test) is equal to Production and in an appropriate state?

In Skype we were facing multiple challenges associated with database testing:
- Simplifying complex Production architecture of thousands of PostgreSQL instances, interconnected with RPCs and replications, infrastructure servers and external DB scripts, into their Test counterparts.
- Constantly growing hardware requirements, insufficient cleanup of data generated in Test.
- Differences between Test and Production were appearing and accumulating. Recognizing and fixing them required lots of effort.

Query optimization is an important problem, which solution has a great influence on DBMS performance, especially for complex queries. In this talk we consider PostgreSQL query optimizer and specifically cardinality estimation problem for correlated clauses, which is one of the most well-known drawbacks of query optimizers in general. In the talk we propose our solution for this problem which involves machine learning methods and is available for PostgreSQL 9.6 as an extension with a patch. We discuss the experimental evaluation, advantages, disadvantages, and fields of application of the proposed approach as well.

VIDEO