PgConf.Russia 2017

Currently, to execute SQL queries PostgreSQL uses interpreter, which implements Volcano-style iteration model. At the same time it’s possible to get significant speedup by dynamically JIT-compiling query “on-the-fly”. In this case it’s possible to generate code that is specialized for given SQL query, and perform compiler optimizations using the information about table structure and data types that is already known at run time. This approach is especially important for complex queries, which performance is CPU-bound.

Input-output (IO) performance issues have been on DBAs’ agenda since the beginning of databases. The volume of data grows rapidly and time is of an essence when one needs to get necessary data fast from the disk and, more importantly, to the disk.

For most databases it is relatively easy to find checklist of recommended Linux settings to maximize IO throughput and, in most cases, this checklist is indeed good enough. It is however essential always to understand how the optimisation of those settings actually works, especially, if you run into corner cases.

It's not a secret that PostgreSQL connections are expensive so you should save them. To solve this problem there are PgPool-II and PgBouncer for quite a long time. At Yandex tens of thousands of connections to a single database is not a surprise so we use pgbouncer since time immemorial. This talk gives an overview of problems we faced and ways to solve them.

VIDEO

Nowadays one can make a decent urban research based simply on public datasets, making interesting and unexpected insights. In the presentation, I'll show examples of these calculations in PostGIS, the industry standard de-facto.

But just PostGIS is not enough. You need tools to import, verify and visualize the data. It's critically important to visualize the data live, to debug your calculations and shorten iterations. I'll describe all these steps:

1. Collecting the data: public API, OpenStreetMap; direct user input.
2. 3rd party APIs for calculations.
3. Visualization of GIS and other sorts of data: QGIS, Matplotlib, Zeppelin integrated with PostGIS.
4. Debugging the calculations: live visualization (Arc, QGIS, NextGIS Web)
5. Scripting and minimizing the chores: Makefile, Gulp