Recovery use cases for Logical Replication in PostgreSQL 10
Avito is the biggest classified site of Russia, and the third largest classified site in the world (after Craigslist of USA and 58.com of China). In Avito, ads are stored in PostgreSQL databases. At the same time, for many years already the logical replication is actively used. With its help, the following issues are successfully solved: the growth of data volume and growth of number of requests to it, the scaling and distribution of the load, the delivery of data to the DWH and the search subsystems, inter-base and internetwork data synchronization etc. But nothing happens "for free" - at the output we have a complex distributed system. Hardware failures can happen - it is natural - you need to be always ready for it. There is plenty of samples of logical replication configuration and lots of success stories about using it. But with all this documentation there is nothing about samples of the recovery after crashes and data corruptions, moreover there are no ready-made tools for it. Over the years of constantly using PgQ replication, we have gained extensive experience, rethought a lot, implemented our own add-ins and extensions to restore and synchronize data after crashes in distributed data processing systems. In this report, we would like to show how our experience can be shifted to a new logical replication subsystem in 10th version of PostgreSQL. In the current implementation, these are only non-trivial solutions - there is a number of issues for the community, that come down to implementing simple recovery mechanisms - as simple as configuring the replication in 10th version.
Слайды
Видео
Другие доклады
-
22 мин
Dmitry Cremer МИА "Россия Сегодня"Cases for monitoring & troubleshooting at highload PosgreSQL DB
A database is one of the key components in any information system, requiring the monitoring of multiple metrics. The talk highlights examples and approaches of monitoring and analysis of PostgreSQL performance that allow to minimize the load on the database server from the monitoring and data collection system for the subsequent analysis of problem situations.
- Quantum effects or as an observer affect the observed system
- Features of collecting metrics while monitoring the database with Zabbix
- Data collection for analytics and visualization PostgreSQL queries with rsyslog + kafka + clickhouse + grafana
- Operational Analysis Tools for DB loglile
-
90 мин
Ivan Frolkov Postgres Professionalpgpro_scheduler and cryptocurrency transactions
Apart from its main purpose of scheduling tasks, pgpro_scheduler can also deal with chained transactions. It can be used in various scenarios of asynchronous data processing.
This tutorial demonstrates pgpro_scheduler features that ensure secure processing of chained transactions. We'll be using cryptocurrency transactions as an example.
pgpro_scheduler is included into Postgres Pro Enterprise as an extension.
-
45 мин
Dmitriy Sarafannikov ЯндексHow to save statistics during major update, and what can be the consequences
It's not a secret for anyone that statistics can not be transferred with a major upgrade. For small and not heavily loaded databases this is not a problem, you can quickly collect new statistics. But we have databases with a volume of about 5TB and a load of about 100k rps, for which it became a big problem: taking off without statistics, the replicas could not even replay WAL. In my report I'll tell you what tricks we went to upgrade these databases with requirements of 100% read only availability, about what mistakes were made, and about how these errors were painfully corrected. The result of these errors was the extension called "pg_dirty_hands", in which we will collect various hacks, which can be last resort to repair data corruption.
-
90 мин
Gregory Smolkin Ozon