title

text

W
Wiktor Brodło Adjust GmbH
: December
45 мин

Bagger: How we migrated 1 PB of data from Elasticsearch into PostgreSQL

In this talk, I will tell you the story of how a bunch of sysadmins got sick of having to resuscitate their petabyte-sized Elasticsearch cluster and decided to replace it with some tried technologies: PostgreSQL, Kafka, a bit of Redis, lots of glue, and the typical sysadmin stubbornness. The result is Bagger: the sysadmin answer to Big Data. A fast, fairly reliable, fault-tolerant store, used mostly for logging timestamped events for some amount of time. Bagger is named the Bagger series of bucket-wheel excavators, feats of German engineering and some of the largest land vehicles ever produced by man. Just like the excavators that dig through tons of material, our Bagger digs through tons data.

Слайды

Видео

Другие доклады

  • Kirill Borovikov
    Kirill Borovikov ООО "Компания "Тензор"
    45 мин

    Explain.sbis.ru as a bulk query optimization tool

    How to optimize query processing in PostgreSQL? What if we deal with hundreds of servers and thousands of instances? The company "Tensor" has developed a special tool - explain.sbis.ru, which enables synchronous collection and analysis of queries, visualization of implementation plans, and monitoring of bugs in database.

  • Ivan Frolkov
    Ivan Frolkov Postgres Professional
    90 мин

    pgpro_scheduler and cryptocurrency transactions

    Apart from its main purpose of scheduling tasks, pgpro_scheduler can also deal with chained transactions. It can be used in various scenarios of asynchronous data processing.

    This tutorial demonstrates pgpro_scheduler features that ensure secure processing of chained transactions. We'll be using cryptocurrency transactions as an example.

    pgpro_scheduler is included into Postgres Pro Enterprise as an extension.

  • Alexander Korotkov
    Alexander Korotkov Postgres Professional
    45 мин

    Pluggable storages

    Pluggable storages is hot subject in PostgreSQL development. The period of heated debates about whether we need them is over. Skepticism about pluggable storages, based on concern that they may be source of inconsistent behavior, was weakened after criticism of PostgreSQL MVCC implementation from Uber side. It became widely understandable that pluggable storages are needed at least for an alternative MVCC implementation. And that is one of way-points for pluggable storages interface design.

    At the moment, work on pluggable storages is in the practical stage. There is a thread is pgsql-hackers where few people are developing patchset and several people are doing review.

    This talk will cover following subjects:

    • overview of pluggable storages interface;
    • changes in PostgreSQL core required to implement this interface;
    • current and potential implementations of pluggable storages including heap with undo-log and in-memory OLTP engine;
    • current state of patchset and prospective of its commit
    • further development of interface allowing more possibilities in pluggable storages (columnar, index-organized, LSM and so on).

  • Egor Rogov
    Egor Rogov Postgres Professional
    90 мин

    Tutorial: More indexes, good and various

    "And telling GIN from SP-GIST was quite beyond his wit, we found", said the classic. Can you? This masterclass is about not-so-often used index types (compared to conventional B-tree) which however can do a great job for you. We will look into internal mechanics of these indexes and discuss cases where they can be successfully applied. Also we will talk about some peculiarities of PostgreSQL index access. To spend time efficiently, listeners are required to have basic knowledge of PostgreSQL and should be used to read plans of simple queries.

    Materials of the master class

    Backup copy of the database with demo data can be downloaded here: