PGConf.Russia 2018

A database is one of the key components in any information system, requiring the monitoring of multiple metrics. The talk highlights examples and approaches of monitoring and analysis of PostgreSQL performance that allow to minimize the load on the database server from the monitoring and data collection system for the subsequent analysis of problem situations.

• Quantum effects or as an observer affect the observed system
• Features of collecting metrics while monitoring the database with Zabbix
• Data collection for analytics and visualization PostgreSQL queries with rsyslog + kafka + clickhouse + grafana
• Operational Analysis Tools for DB loglile

In this talk, I will tell you the story of how a bunch of sysadmins got sick of having to resuscitate their petabyte-sized Elasticsearch cluster and decided to replace it with some tried technologies: PostgreSQL, Kafka, a bit of Redis, lots of glue, and the typical sysadmin stubbornness. The result is Bagger: the sysadmin answer to Big Data. A fast, fairly reliable, fault-tolerant store, used mostly for logging timestamped events for some amount of time. Bagger is named the Bagger series of bucket-wheel excavators, feats of German engineering and some of the largest land vehicles ever produced by man. Just like the excavators that dig through tons of material, our Bagger digs through tons data.

Patroni is a Python application to create high-availability PostgreSQL clusters based on the streaming replication. It is used by Red Hat, IBM Compose, Zalando and many other companies. This tutorial will highlight Patroni architecture, provide attendees with hands-on experience of configuring high-availability PostgreSQL clusters with Patroni, describe how to take advantage of numerous additional features and give an opportunity to learn more about common mistakes related to running Patroni and its troubleshooting.

In order to take most out of the Patroni tutorial one needs a laptop with git, vagrant and virtual box installed.

Vagrant can be obtained from https://www.vagrantup.com Virtualbox is at https://www.vagrantup.com

Alternatively, one can install your Linux distribution packages (or use homebrew on Mac).

Once Vagrant and Virtualbox are installed one can run the Patroni VM by issuing the following commands:

$ git clone https://github.com/alexeyklyukin/patroni-training
$ cd patroni-training
$ vagrant up

When the setup concludes Patroni box can be accessed via ssh using vagrant ssh command.

WAL-G is simple and effective disaster recovery tool for PostgreSQL using cloud storages. In its core functionality, WAL-G is the successor of WAL-E rewritten in Go. But there is one new neat feature - delate-backups. WAL-G delta-backups, whenever possible, stores only pages, changed since the previous backup. In this talk, I'm going to describe development process of this feature.

Surprisingly, most important and complicated question was the design of the interface: WAL-e is simple and comprehensive, keeping these properties was goal #1. Technical details of implementation were covering some underwater stones too. Besides these, I want to discuss the perspective of technological development and future coordination of recovery tools developers.