PgConf.Russia 2020

From the very beginning, PostgreSQL in Avito has been solving very important tasks. All the main architecture components are built around DBMS. For over 10 years, the project has been actively evolving, and the infrastructure and architecture have changed a lot.

The talk will start with an overview of how PostgreSQL infrastructure and architecture have advanced in Avito over time and which challenges have been successfully resolved.

Then we'll discuss PostgreSQL usage scenarios in Avito in 2020: microservice architecture, sharding, hosting multiple databases on a single server instance, DBaaS (Database discovery, access control, failover, backup, archive, resource sharing, etc.), integration, and team evolution.
And finally, I'll share our backlog and wishlist.

Many businesses which use Database management systems like Oracle, DB2 & MS SQL are unreliable these days. Moreover, the costs incurred in maintaining these systems and its product licenses keeps on increasing. As the competitors are migrating over to the new technologies and tools available in the market, it is necessary for these businesses to migrate to new environment which is efficient, consistent and reliable to stay in the market and the technologies used in the current environment have become obsolete or no longer serve the business purpose. PostgreSQL has emerged as a top open-source RDBMS software. Since there is no licensing cost associated with it most of the companies are planning to migrate the databases which are currently running on other RDBMS like Oracle, DB2, MS SQL server to PostgreSQL. This report summarizes the various methodologies, procedures and techniques involved in successfully migrating the data from Oracle to PostgreSQL & DB2 to PostgreSQL. Migration is not a simple effort there should be proper planning and testing involved in this right from database connectivity to performance analysis. In this paper we are going to cover most of the steps which we need to consider before the migration and after the migration like choosing the correct tools for implementing the migration, time taken to migrate ,data compatibility, code conversion, application connectivity to database, database configuration parameters, performance analysis, replication setups, database monitoring, patching and backup strategies.

I will share our experience of migrating a server application for Russian railways from Oracle 11g Standard Edition to vanilla PostgreSQL 11.5.
At the time of migration, the database contained about 200 stored procedures of over 60000 lines of Oracle PL/SQL code (which has been developed since 2006, that is, for more than 12 years), about 250 tables, and 50GB of data.
Starting with a prologue, we'll describe our adventures along the migration process, as well as pleasant and unpleasant surprises we encountered, and finally get to an epilogue and a happy end. The story is told on behalf of an Oracle user exploring Postgres.

Cloud storage has some unique characteristics compared to traditional storage mainly because it is virtualized and controlled by software. One example is that AWS EBS shows higher throughput with larger I/O size up to 256 KiB without hurting latency. Hence, a user can get only about 4 MiB/sec with 1,000 IOPS EBS volume if the I/O request size is 4 KiB, whereas a user can get about 250 MiB/sec if the I/O request size is 256 KiB. This is because EBS consumes one I/O in a given IOPS budget for every I/O request regardless of the I/O size (up to 256 KiB). Unfortunately, PostgreSQL cannot exploit the full potential of cloud storage because PostgreSQL has designed without considering the unique characteristics of cloud storage.

In this talk, I will introduce the AppOS extension that improves the throughput of a write-intensive workload by 10x by transparently making PostgreSQL cloud storage-native. AppOS works like a storage driver that efficiently exploits the characteristics of cloud storage, such as I/O size dependency to storage throughput and latency, atomic write support in cloud block storage, and fast, but non-durable local SSDs. To do this, AppOS comprises a Linux-compatible file I/O stack including virtual file system, page cache, block I/O layer, cloud storage driver. On top of the file I/O stack, syscall module supports registering pre- and post-handler for file I/O-related system calls in order to transparently work without modifying PostgreSQL codes.

I will focus on presenting key use cases and performance results of the AppOS extension after explaining the internals. Specifically, I will show the performance results of OLTP and some batch workloads using standard benchmarking tools like pgbench and sysbench. I will also present performance results and implications on multiple clouds including AWS, GCP, and Azure.