PgConf.Russia 2017

When working with a complex business logic, you often have to implement a workflow - a sequence of several processing steps, with each step implementing a separate part of the business logic. This is usually done with specialized queues, but if there are high reliability demands, it makes sense to do everything on PostgreSQL.

I will describe the tasks that require a workflow implementation, offer a solution, compare it with other options, and tell you about the implementation traps and pitfalls.

The talk will describe how we have implemented storage of large tables (+1 billion rows per day). The project exists in production 2 years. The total amount of data - 300 Tb (25 PostgreSQL servers * 2 Data Center). I'll tell about mistakes in organization of large tables storage in the initial phase of the project, and how these mistakes were corrected. I'll also talk about how to organize the data rotation and archiving. I voiced questions about what we were missing in PostgreSQL 9.4 out of what appeared in the 9.5 and 9.6. And also, what new features we are waiting for new releases of PostgreSQL.

Practical experience of carrying out import substitution with using PostgreSQL in government information system including not only the free software, but also the Russian hardware (Elbrus servers and other).

VIDEO

Database systems are increasing in size and so is the need to process huge amounts of data in real time. As commercial database vendors are bragging about their capabilities we decided to push PostgreSQL to the next level and exceed 1 billion rows per second to show what we can do with Open Source. To those who need even more: 1 billion rows is by far not the limit - a lot more is possible. Watch and see how we did it.

VIDEO