PgConf.Russia 2020

Open-source maintainers face many challenges as projects grow. How to write more required features, fix more issues and have time to watch more pull requests? On the example of WAL-G(backup tool for PostgreSQL) I will tell you about how we solved these problems by launching a course of Open-source development at Ural Federal University, what we achieved and what will we do next.

When migrating data from one DBMS to another, the question arises: choose a third-party tool or to program the migration yourself? Companies, trying to grow competencies within themselves, choose the second option. And they come across the "invention of their own bicycles". However, the market has powerful free data migration tools. One such tool is Pentaho Data Integration, part of the Pentaho Community Edition. The report will discuss the use of this package for data migration between Oracle and PostgreSQL. Particular attention will be paid to the problems with using this tool, and to the tasks of testing for the completeness and integrity of migrated data.

Small video illustration:

Moscow public transport vehicles when moving report their coordinates via GLONASS. Collected data is used for various analyses including timetable development, bottlenecks detection and planning the bus lanes. Until recently we used the PostGIS extension for this purpose but now we are switching to a new PG extension — MobilityDB — designed especially for geodata time series processing. I have compared the table size and the performance of our solution without and with MobilityDB and happy to present the results.

If you care about Postgres performance, there are a number of hardware acceleration options to help with different use cases. Intel Optane DC persistent memory creates new tier in data hierarchy allowing developers to utilize performance of traditional memory combining with volume and persistency of block storage devices. Unlike traditional DRAM-only in-memory systems, where memory is small, expensive, and volatile, Intel Optane DC persistent memory makes it possible to run larger Postgres databases (terabytes) in memory for higher performance. FPGAs are integrated circuits that can be reprogrammed dynamically to accelerate a specific workload such as SQL execution and data compression. FPGA accelerators extend Postgres with hundreds of SQL reader and writer processes that work in parallel on the FPGA. It’s similar to adding hundreds new cores to boost parallel processing on your server.