PGConf.Russia 2021

Data is getting bigger, disks are getting faster, the DBMS optimizer is getting smarter, but the problem of " bloating " remains extremely relevant. I want to share my experience and approach to dealing with this effect on a large number of tables and data in them.

When PostgreSQL was at its formation stage, its transaction ID was chosen to be 32-bit. Back then, no one could imagine that someday we may need more than 4 billion transactions. However, ongoing technical progress and digitization pushed some Postgres instances towards their transaction ID limit. The Postgres community reacted to this with a wraparound of the transaction ID counter. However, constantly growing data volumes exposed PostgreSQL to new challenges. In my presentation, I will cover these challenges and explain how they can be solved with a 64-bit transaction ID, what the consequences will be like, why now it is a good time to implement 64x IDs, and why this hasn't been done previously.

I will not teach you database design or query optimization. This talk will cover Python drivers, PostgreSQL usage in Python, and various tools from sync and async world enabling such usage, and how this ecosystem evolved. We'll discuss Python libraries including aiopg, psycopg2, asyncpg, pgbouncer and how they interact with Postgres.

Many application developers prefer not to have logic on the DB side (in functions) as there are no suitable software development tools, especially for development teams. In the plv8 case, the situation worsens as this function contains both SQL and JavaScript code, and popular IDEs have no support for such a symbiosis. At this tutorial, I will introduce my development named "Plv8 Framework", which considerably simplifies the creation of code on plv8.

The gist is as follows: the JS code that is executed on the DB side can run on the developer's local computer by using node.js, which works on the same v8 interpreter. The plv8.execute special function is replaced by a function from the pg-native npm library, which calls the outer DBMS. I will demonstrate a set of tools chosen by me that enables the following:
- writing JS code in the IDE you prefer and enjoy syntax highlight;
- code debugging in real time (with breakpoint, watch, etc.);
- writing of auto-tests (unit-tests), with a variety of options: Postgres, SQLite, mocks;
- deploying your code in the DBMS;
- usage of additional npm packages (the issue is that all code of the plv8-based function should be included in the function's body, in one file).

You can use this tool regardless of the programming language that you use for the backend. However, it becomes more flexible if you use languages with static typing (like Java, C#, etc.). For the tasks where the backend is an intermediate layer between the frontend and DBMS, logic (or its part) can be placed in plv8/js with dynamic typing, which will simplify the development process.

In addition to the development of new functions on plv8, the framework provides a set of ready-made functions for CRUD operations. These functions are universal, they aren't tied to a certain database structure, and they can work in any project. If you use them, you can do less backend development, in some projects to a very significant extent.

The installation of the plv8 extension is the most complicated part of working with it. However, I have good news: my colleagues prepared Docker files and Docker images for PostgreSQL 13 with pre-installed plv8! So your start with plv8 will be super simple: you need to deploy the Docker container using just one command!
Docker file: PostgreSQL 13 + plv8 v2.13.15
Demo project for you to participate in the tutorial
For the tutorial, you need to install the following:
Node.js (the most desirable is LTS)
IDE for JS (i.e., free Visual Studio Code)
GraphQL Playground