title

text

W
Wiktor Brodło Adjust GmbH
10:00 07 February
45 мин

Bagger: How we migrated 1 PB of data from Elasticsearch into PostgreSQL

In this talk, I will tell you the story of how a bunch of sysadmins got sick of having to resuscitate their petabyte-sized Elasticsearch cluster and decided to replace it with some tried technologies: PostgreSQL, Kafka, a bit of Redis, lots of glue, and the typical sysadmin stubbornness. The result is Bagger: the sysadmin answer to Big Data. A fast, fairly reliable, fault-tolerant store, used mostly for logging timestamped events for some amount of time. Bagger is named the Bagger series of bucket-wheel excavators, feats of German engineering and some of the largest land vehicles ever produced by man. Just like the excavators that dig through tons of material, our Bagger digs through tons data.

слайды

Видео

Другие доклады

  • Дмитрий Шитов
    Дмитрий Шитов ООО "ЦТП"
    22 мин

    How I Met Your Linux

    What is a real cost of not paying for Windows for 1C-user? Is there life without COM? Addressing and other issues for the bunch of PostgreSQL. Scheduling disk resources. How to overcome OS CentOS crash.

  • Alexander Korotkov
    Alexander Korotkov Postgres Professional
    45 мин

    Pluggable storages

    Pluggable storages is hot subject in PostgreSQL development. The period of heated debates about whether we need them is over. Skepticism about pluggable storages, based on concern that they may be source of inconsistent behavior, was weakened after criticism of PostgreSQL MVCC implementation from Uber side. It became widely understandable that pluggable storages are needed at least for an alternative MVCC implementation. And that is one of way-points for pluggable storages interface design.

    At the moment, work on pluggable storages is in the practical stage. There is a thread is pgsql-hackers where few people are developing patchset and several people are doing review.

    This talk will cover following subjects:

    • overview of pluggable storages interface;
    • changes in PostgreSQL core required to implement this interface;
    • current and potential implementations of pluggable storages including heap with undo-log and in-memory OLTP engine;
    • current state of patchset and prospective of its commit
    • further development of interface allowing more possibilities in pluggable storages (columnar, index-organized, LSM and so on).

  • Kirill Borovikov
    Kirill Borovikov ООО "Компания "Тензор"
    45 мин

    Explain.sbis.ru as a bulk query optimization tool

    How to optimize query processing in PostgreSQL? What if we deal with hundreds of servers and thousands of instances? The company "Tensor" has developed a special tool - explain.sbis.ru, which enables synchronous collection and analysis of queries, visualization of implementation plans, and monitoring of bugs in database.

  • Valery Kosarev
    Valery Kosarev -
    22 мин

    Pluggable storage for large objects

    Storing binary data in database tables is sometimes a good solution for a particular project. But sometimes, due to changes in conditions or insufficient consideration of decisions, such storage is becoming a real nightmare. If there is an understanding of how and where to place these data, the transition to the new solutions are often very hard, often require modification in the application code and downtime the system for migration. The presentation is a particular solution of such problems. Our extension allows to move binary data from database to the storage Ceph and not only. And does it seamless for the applications.