PGConf.Russia 2018

It's not a secret for anyone that statistics can not be transferred with a major upgrade. For small and not heavily loaded databases this is not a problem, you can quickly collect new statistics. But we have databases with a volume of about 5TB and a load of about 100k rps, for which it became a big problem: taking off without statistics, the replicas could not even replay WAL. In my report I'll tell you what tricks we went to upgrade these databases with requirements of 100% read only availability, about what mistakes were made, and about how these errors were painfully corrected. The result of these errors was the extension called "pg_dirty_hands", in which we will collect various hacks, which can be last resort to repair data corruption.

For over 15 years JetBrains have strived to make effective developer tools. IntelliJ IDEA for java is the most popular of them. IntelliJ supports databases as well, and finally we in JetBrains decided to bring this functionality and our experience in programming languages to the world of SQL. That’s how DataGrip appeared. In my talk I will tell how DataGrip helps developers by automating routine checks and corrections and speeds up developer’s production. I will also show how to extend IDE’s functionality and try to understand which actual problems we can solve in future.

Last year we announced Skala-SR / Postgres Pro database machine with a hardware and software support for remote direct memory access (RDMA) as a key feature. The first appliances have been already installed at customer sites, and even with the first version customers got some constructions, that were impossible without RDMA and CPU offload (got with Mellanox networking). However, capabilities of this equipment are much wider, and this talk is dedicated to current works and prospective developments for this topic.

In the last six months I have been working with a massive OLAP environment with 20TB shards, spanning around 400TB of data. Come to listen to how we make it all work, the challenges, and the skills involved. This talk has very little in common with the 10TB and Beyond talk because the data environments are very different.

We will cover analytics performance, data alignment, reasons for building extensions in C, and moving data around between servers in multiple data centers.