PgConf.Russia 2020

Today no one is surprised by cloud infrastructure anymore, but not all its components are easy to deploy in cloud. For example, the database is always very demanding in terms of performance and resources. Scaling and fault tolerance are the most acute problems, that's why we have been observing rapid development of alternative DBMS in the recent years. However, traditional relational DBMS have already accumulated a lot of various features, so they often remain the first choice. Besides, they are constantly evolving and offer a wide variety of scaling tools. I will mainly speak about PostgreSQL, when you should consider scaling, and how to do it right.

We will touch upon the following topics:
- Streaming replication and balancing read/write workloads
- Logical replication and data sharding
- High availability and fault tolerance

This talk should be interesting to DBAs, system administrators, team leads, infrastructure architects, as well as wider audience dealing with PostgreSQL.

In my talk, I will tell how we built a geographically distributed system of personal data storage based on Open Source software and PostgreSQL. The concept of the inCountry business is to provide customers with a ready-to-use infrastructure for personal data storage. Our business customers are ensured that their customer’s personal data is securely stored within their country’s borders. We wrote an API and SDK and built a variety of services. Our system complies with generally accepted security standards (SOC Type 1, Type 2, PCI DSS, etc.). We built our infrastructure with Consul, Nomad, and Vault, used PostgreSQL, ElasticSearch as a storage system, Nginx, Jenkins, Artifactory, other tools to automate management and deployment. We have assembled our development and management teams - DevOps, Security, Monitoring, and DBA. We use both cloud providers and bare-metal servers located in different regions of the world. Development of the system architecture and ensuring the stability of the infrastructure, consistent and secure operation of all its components is the main task facing our teams.

If you care about Postgres performance, there are a number of hardware acceleration options to help with different use cases. Intel Optane DC persistent memory creates new tier in data hierarchy allowing developers to utilize performance of traditional memory combining with volume and persistency of block storage devices. Unlike traditional DRAM-only in-memory systems, where memory is small, expensive, and volatile, Intel Optane DC persistent memory makes it possible to run larger Postgres databases (terabytes) in memory for higher performance. FPGAs are integrated circuits that can be reprogrammed dynamically to accelerate a specific workload such as SQL execution and data compression. FPGA accelerators extend Postgres with hundreds of SQL reader and writer processes that work in parallel on the FPGA. It’s similar to adding hundreds new cores to boost parallel processing on your server.

At the master class, we will learn how to execute fast load tests of Postgres databases: optimizing database configurations, data structure, indexes, OS settings, etc. To do this, we will create a code, build the infrastructure for the test from it and will do the test. As a result, we get a flexible tool in the code to which you can attach any monitoring, and for which you don't have to pay a lot of money, because the environment is created in 7 minutes in an empty AWS account and destroyed after test