PGConf.Russia 2018

In this talk, I will tell you the story of how a bunch of sysadmins got sick of having to resuscitate their petabyte-sized Elasticsearch cluster and decided to replace it with some tried technologies: PostgreSQL, Kafka, a bit of Redis, lots of glue, and the typical sysadmin stubbornness. The result is Bagger: the sysadmin answer to Big Data. A fast, fairly reliable, fault-tolerant store, used mostly for logging timestamped events for some amount of time. Bagger is named the Bagger series of bucket-wheel excavators, feats of German engineering and some of the largest land vehicles ever produced by man. Just like the excavators that dig through tons of material, our Bagger digs through tons data.

MQTT is an effective data exchange protocol for IOT devices. Based on own modified EMQTT plug-in, the IoT project architecture uses PostgreSQL as a central processing and storing system for data coming from sensors in real time. The report will provide an example of the IoT hardware and software platform solution, based up on the MQTT protocol, where PostgreSQL is responsible for key functionality, providing processing, collecting and storage of data from a distributed network of IoT devices.

In this talk I will share the experience of migration, NoSQL data to a relational view convert, and tell you how we managed to speed up the application in 2 times.

Initially, we used PosgtgreSQL and MongoDB to store all the data. In practice, we found out that the separation is extremely inconvenient. We spent a lot of time and attention on it.

My talk will describe how we moved the data from MongoDB to PostgreSQL using mosql. As a result, all the data can be received by one query now, and the table schema ensures data consistency.