title

text

Peter  van Hardenberg
Peter van Hardenberg Heroku
10:00 05 February
45 мин

Megascale PostgreSQL-as-a-Service: Operating 10^6 Databases

Heroku Postgres is a cloud database service and the largest provider of PostgreSQL as a service anywhere. We operate more than 1,000,000 PostgreSQL databases with a team of about 10 people. We may be the most efficient DBAs in history, with approximately 100,000 databases per person on our team! This talk will introduce the opportunity and challenges of building and operating a cloud database service, as well as discussing the strategies we use to build, operate, and scale this product and team for the last six years now. We will include details about * a brief introduction to the service to provide context * strategies to design and build such a data service * operational war stories like how to recover from losing thousands of servers at once, * common challenges users have with Postgres * and a basic overview of the technical architecture

This is a complementary talk to Will Leinweber's talk, which will go into much more depth on the architecture of the software we have written.

Слайды

Видео

Другие доклады

  • Lev Laskin
    Lev Laskin Электрон
    45 мин

  • Alex Chistyakov
    Alex Chistyakov ISST Lab, ITMO University
    22 мин

    Stainless steel elephant: PostgreSQL performance testing continued

    We love to stress test software, since we are a performance engineering company. Our friends from a hosting company servers.com provided us with a modern dedicated server so we immediately started to test PostgreSQL in different environments, including SmartOS, DragonFly and Windows. We would like to present our results (and all the gory details) to community.

  • Alexander Korotkov
    Alexander Korotkov Postgres Professional
    45 мин

    PostgreSQL extendability: Origins and new horizons

    Postgres was initially designed to support access methods extendability. Well known citation about access method in Postgres claims: "It is imperative that a user be able to construct new access methods to provide efficient access to instances of nontraditional base types" Michael Stonebraker, Jeff Anton, Michael Hirohama. Extendability in POSTGRES, IEEE Data Eng. Bull. 10 (2) pp.16-23, 1987

    Initially, heap was just one for access methods. So, extendability of access methods would also mean pluggable storage engines in modern terms. For now, only index access methods are defined in pg_am table of system catalog. Those index access methods also have well-defined interface. Therefore in order to meet initial design PostgreSQL need to support two features:

    • Pluggable index access methods, i.e. ability to implement new index types by adding new tuples to pg_am;
    • Pluggable storage engines, i.e. ability to implement completely different storages for tables without traditional heap.

    Besides mechanical work like "CREATE ACCESS METHOD" command, extensible index access methods needs to be WAL-logged. For now, community doesn't want extensions to define their own WAL-records, because there is a chance to break both recovery and replication, which is not acceptable. Another approach is to define generic WAL-records, that specify a difference between pages in generalized way.

    There are only few DBMS which support pluggable storage engines now. MySQL is the most common example here. However, dealing with different storage engines in MySQL is like dealing with different DBMS. This is not the way PostgreSQL should go from our view.

    However, now PostgreSQL users realize benefits from other storages. Ideas of columnar storages and in-memory storages for PostgreSQL are very popular. Simultaneously, technical possibilities to implement them are growing. FDW and custom nodes are arrived. Generic WAL and extensible index access methods are pending for 9.6. Much work in the direction of pluggable storage engines is already done even if it had different aims.

    It's time for PostgreSQL core developers to think about native support of pluggable storages without kludges. Finally, we should get "CREATE STORAGE ENGINE name ..." command as legal extendability mechanism.

    In this talk we will show current state on pluggable index access method and design of pluggable storage engines.