PgConf.Russia 2017 talks

Towards 1M writes/sec: Scaling PostgreSQL using Citus MX

Marco Slot
Citus Data, Principal Engineer

Marco has an M.S. in Parallel and Distributed Computer Systems from VU University Amsterdam and a PhD on cooperative self-driving cars at Trinity College Dublin. While studying, he also worked as a software engineer on Amazon CloudFront and Route 53 for several years, driving the design of latency-driving request routing and several other key features. He now works as a software engineer on Citus Data.

Citus allows you to distribute postgres tables across many servers. It extends postgres to transparently delegate or parallelise work across a set of worker nodes, enabling you to scale out the CPU and memory available for queries.

One year ago, we began a long journey to allow Citus to scale out another dimension: write throughput. With writes being routed through a single postgres node, write throughput in Citus was ultimately bottlenecked on the CPUs of a single node. Citus MX is a new edition of Citus which allows distributed tables to be used from from any of the nodes, enabling NoSQL-like write-scalability. We will discuss the main technical challenges we had to overcome such as DDL replication, cluster-wide locking, failure recovery, and node additions. We will also show how we use a dynamically changing pgbouncer set-up to achieve linear scale and the performance results we have achieved so far.

Finally, we will conclude with a live demo of using Citus MX to handle an IoT workload.

VIDEO

Slides