PgConf.Russia 2017

The talk will describe how we have implemented storage of large tables (+1 billion rows per day). The project exists in production 2 years. The total amount of data - 300 Tb (25 PostgreSQL servers * 2 Data Center). I'll tell about mistakes in organization of large tables storage in the initial phase of the project, and how these mistakes were corrected. I'll also talk about how to organize the data rotation and archiving. I voiced questions about what we were missing in PostgreSQL 9.4 out of what appeared in the 9.5 and 9.6. And also, what new features we are waiting for new releases of PostgreSQL.

This report overviews some ideas and implementations to speedup different parts of generalized search trees (GiST): 1. Intrapage indexing 2. Fractal tree technology 3. Modern algorithms for spatial indexing (RR*-tree) 4.. Possible advancements of GiST API

VIDEO

Database sharding enables a distribution of the database over a large number of machines, greatly improving performance. With the advent of Foreign Data Wrappers (FDW), it's possible to consider a database sharding in PostgreSQL with acceptable level of code changes using FDW. We've been working on enhancing around FDW infrastructure such as foreign table inheritance and pushing down so that PostgreSQL can execute the distributed query efficiently using FDW. In this talk, I'll cover what FDW-based sharding is and what use-cases it can cover. And then I'll demonstrate how to build sharding and describe our achievement of a FDW-based sharding in PostgreSQL community. Finally, I'll describe further enhancements to FDW such as Async Execution and Distributed Transaction Support.

Partitioning is a long-awaited feature in PostgreSQL. Although Postgres supports partitioning via inheritance, this approach has some disadvantages, such as the need to manually create partitions and support triggers, significant planning overhead, and no query execution optimizations. In this talk, we’ll tell you about the pg_pathman extension we are developing. pg_pathman supports HASH and RANGE partitioning, performs planning and execution optimizations, supports fast insert by using Custom Node instead of triggers, provides functions for partition management (add, split, merge, etc.), supports FDW, non-blocking data migration, and more. We'll also speak about pg_pathman integration with Postgres Pro Enterprise Edition and Oracle-like syntax support for partitioning. Finally, we'll discuss new partitioning capabilities in PostgreSQL 10, the already implemented features and further development plans.

VIDEO