PgConf.Russia 2017 Conference Program

Modern state of backup tools for PostgreSQL certainly leaves some room for improvements. Standard utilities provides only basic functionality, third-party tools solve some, but not all, problems. To take backups reliably, fast, and incrementally on page level, one needs not only a backup tool, but also some support from the database server. On this tutorial we will talk about our new backup and recovery manager ProBackup, and will show it in action.

Demonstration script:

• pg_probackup_script.txt

This tutorial is about various applied JSON usage patterns and the related PostgreSQL functionality. We will discuss data storage in the JSON format, retrieving, changing, and searching such data, JSON features for simple SQL queries, as well as using JSON in stored procedures in different languages. You’ll get hands-on experience with some of the discussed problems on the provided virtual machines.

Debugging, profiling, and tracing of the executed commands play an important role in development of any applications. This is also true for developing stored procedures in DBMS.

PostgreSQL offers various tools (both built-in and external) for these purposes.

In this talk, we will provide an overview of the available tools and their advantages and disadvantages, as well as a detailed demo of their use cases.

This talk is a part of a basic course for backend application developers (DEV1), which “Postgres Professional” company plans to announce in the near future.

Nowadays one can make a decent urban research based simply on public datasets, making interesting and unexpected insights. In the presentation, I'll show examples of these calculations in PostGIS, the industry standard de-facto.

But just PostGIS is not enough. You need tools to import, verify and visualize the data. It's critically important to visualize the data live, to debug your calculations and shorten iterations. I'll describe all these steps:

1. Collecting the data: public API, OpenStreetMap; direct user input.
2. 3rd party APIs for calculations.
3. Visualization of GIS and other sorts of data: QGIS, Matplotlib, Zeppelin integrated with PostGIS.
4. Debugging the calculations: live visualization (Arc, QGIS, NextGIS Web)
5. Scripting and minimizing the chores: Makefile, Gulp

This tutorial shows how to properly monitor PostgreSQL. We will discuss the mamonsu utility, see how to configure it, examine its hidden features and learn how to extend them.

Java is one of the most used languages when programming with PostgreSQL databases. Join this tutorial to learn or review the techniques to connect to postgres, best programming practices with JDBC, and to explore jOOQ, a mapper software that allows you to use the full power of SQL and postgres advanced query features while avoiding all the boilerplate code.

This tutorial is very practical: most of the time will be dedicated to iterate through code samples. It will cover:

• Introduction to Java and PostgreSQL
• Ways of connecting to PostgreSQL from Java (not only JDBC!)
• Introduction to JDBC. JDBC types. PostgreSQL JDBC
• Code demo: JDBC with PostgreSQL. From Java 1.4 to Java 8, best practices and code samples
• Code demo: jOOQ, a great mapper for PostgreSQL
• Java inside PostgreSQL
• The future of Java and PostgreSQL

About two-thirds of the tutorial will be dedicated to iterate over code samples and demos. All the code would be available from public open-source repositories and built with maven, so that any attendee may download it and build easily to play with it during the tutorial (although not required).

VIDEO

Part 1

Part 2

Part 3

Part 4

Full text search in PostgreSQL is probably the most advanced one among relational DBMS. This tutorial will explain how to setup full text search configurations and dictionaries and how to build a ful text search system using an example of a simple popular science web site, with demonstration of various ranking functions. Also I will tell about new RUM index, which allows to accelerate execution of some kinds of full text queries and implements a new improved ranking function.

The PostgreSQL community is over 20 years old, but the history of PostgreSQL dates back even farther. In this talk, we'll learn about the roots of the Postgres project, learn about some of the people who contribute to it, study how it has changed over time, and pay special attention to the many contributions of Russian people.

It's not a secret that PostgreSQL connections are expensive so you should save them. To solve this problem there are PgPool-II and PgBouncer for quite a long time. At Yandex tens of thousands of connections to a single database is not a surprise so we use pgbouncer since time immemorial. This talk gives an overview of problems we faced and ways to solve them.

VIDEO

MongoDB is a successful database in the NoSQL space, mostly used for OLTP-type workloads. However, due to the lack of ACID (transactions in particular) and significant performance issues with OLAP/DW workloads, more and more MongoDB users are considering migrating off of MongoDB to a RDBMS, where PostgreSQL is the usual choice. This represents a significant opportunity for the PostgreSQL ecosystem, to "bring NoSQL to SQL". This talk will present the challenges that MongoDB users are facing and the state of the art of the available tools and open source solutions available to perform ETL and live migrations to PostgreSQL. In particular, ToroDB Stampede will be discussed, an open source solution that replicates live from MongoDB, transform JSON documents into relational tables, and stores the data in PostgreSQL.

VIDEO

I want to present a new custom access method, which extends the current GIN capabilities using additional information stored in posting tree/list. For example, positional information as an additional information allows new AM returns results in relevance order, which could considerably improve execution time of full text queries.

Streaming replication has been introduced in 2010 and quickly became one of the most popular features of PostgreSQL. Today, it is hard to imagine PostgreSQL installation without streaming replication. With its stability, high efficiency and ease of configuration one would have thought it is an optimal feature. However, while using it you might sometimes enter murky waters. This often can be resolved by using a combination of built-in and third party troubleshooting tools. In my talk I will provide an overview of these tools and explain how with their help one can diagnose, understand and eliminate problems related to streaming replication. I will also go through the most frequent issues occurring when streaming replication is used and will propose possible solutions. This talk is primarily aimed at DBAs and system administrators who use PostgreSQL in their day-to-day.

VIDEO

This talk is intended for Oracle DBAs whose are considering to start using PostgreSQL as another RDBMS engine in their DB portfolio or for PostgreSQL DBAs, who wants to learn some of similarities and differences between Oracle and PostgreSQL. This talk is not intended to be a migration guideline neither advocating one platform over another. General database structure overview for both platforms will be compared in terms of memory configuration parameters; logical to physical structure as well as we’ll touch some security point of view and backup & recovery tools available. Oracle container databases are not covered. The talk want’s to help create a bridge for DBA and avoid some frustrations at the first Oracle DBA step to discover, learn and start using PostgreSQL as reliable fully functional database.

VIDEO

Database sharding enables a distribution of the database over a large number of machines, greatly improving performance. With the advent of Foreign Data Wrappers (FDW), it's possible to consider a database sharding in PostgreSQL with acceptable level of code changes using FDW. We've been working on enhancing around FDW infrastructure such as foreign table inheritance and pushing down so that PostgreSQL can execute the distributed query efficiently using FDW. In this talk, I'll cover what FDW-based sharding is and what use-cases it can cover. And then I'll demonstrate how to build sharding and describe our achievement of a FDW-based sharding in PostgreSQL community. Finally, I'll describe further enhancements to FDW such as Async Execution and Distributed Transaction Support.

Tencent, based in China, is one of the world's largest companies in the social networking space. This talk discusses how Tencent modified the code of Postgres-XC to meet their internal payment system requirements

In the modern world, an increasing number of IT companies are moving their resources to the cloud and Zalando is not an exception. A rapid growth our company is experiencing along with an adoption of microservices were the main driving forces behind the changes introduced into the deployment procedure of new PostgreSQL clusters and the solution of the automatic failover problem. The majority of existing solutions for automatic failover require manual configuration of every cluster instance and complicates provisioning new clusters and new nodes into existing cluster.

Partitioning is a long-awaited feature in PostgreSQL. Although Postgres supports partitioning via inheritance, this approach has some disadvantages, such as the need to manually create partitions and support triggers, significant planning overhead, and no query execution optimizations. In this talk, we’ll tell you about the pg_pathman extension we are developing. pg_pathman supports HASH and RANGE partitioning, performs planning and execution optimizations, supports fast insert by using Custom Node instead of triggers, provides functions for partition management (add, split, merge, etc.), supports FDW, non-blocking data migration, and more. We'll also speak about pg_pathman integration with Postgres Pro Enterprise Edition and Oracle-like syntax support for partitioning. Finally, we'll discuss new partitioning capabilities in PostgreSQL 10, the already implemented features and further development plans.

VIDEO

With big data threads, even the upload of data to a database can often be problematic – apart from the data upload itself, you need to create indexes, perform VACUUM after the upload for correct Index-only scans, etc. From this talk, you will learn how to avoid most of such problems (if not all of them).

VIDEO

The report is focused on the topic of automation of migration to PostgreSQL from other databases using Ispirer Migration and Modernization Toolkit. The issues of data and SQL code migration, conversion of client applications, embedded SQL and database API will be covered, the examples of implemented projects of migration from Oracle to PostgreSQL and from Microsoft SQL Server to PostgreSQL will be given.

VIDEO

Modern high-performance and scale-out Database solutions need to move huge amounts of data between the system components. They need to perform it quickly and without delay for the end-user. While developing interconnect technologies Mellanox focuses on highest bandwidth, ultra-low latency, and advanced CPU offload capabilities. In this session Michael will tell about fast interconnect value for Database and Big Data solutions and will share Mellanox experience in building such systems in cooperation with leading global vendors.

VIDEO

1. 1C:Enterprise 8 and PostgreSQL 9 interoperability 1.1 Changes in new 1C platform versions 1.2 v81c_data and v81c_index schemas 1.3 Sending 1C queries to SQL 1.4 Using 1C technological log events for PostgreSQL diagnostics
2. Analyzing queries that affect PostgreSQL performance 2.1 A free tool for automating log parsing 2.2 Pareto principle in action 2.3 Installation and configuration of the tool 2.4 A case study of query optimization 2.4.1 An issue in a PostgreSQL query 2.4.2 Finding non-optimal operations in a query 2.4.3 Resolving inefficiencies
3. PostgreSQL statistics for performance diagnostics 3.1 Comparing Postgres with MS SQL Server 3.2 Troubleshooting locks 3.3 Operating load diagnostics 4 Case studies by the gilev.ru team

I will tell you about why Postgres is first-choice product as a foundation for your BI system with classical OLAP workload. Briefly it will be said about existing open source BI solutions.

I will also describe specific of our architecture, why we chose snowflake scheme and how we are doing extract, transformation and load procedures. It will be mentioned about special Postgres tuning for OLAP and massive data bulkload workloads. Also I will let you know about Postgres usage as a column database with cstore_fdw by Citus and results achieved. Cons and problems of our approach will be described in the end of the talk.

VIDEO

The talk will describe how we have implemented storage of large tables (+1 billion rows per day). The project exists in production 2 years. The total amount of data - 300 Tb (25 PostgreSQL servers * 2 Data Center). I'll tell about mistakes in organization of large tables storage in the initial phase of the project, and how these mistakes were corrected. I'll also talk about how to organize the data rotation and archiving. I voiced questions about what we were missing in PostgreSQL 9.4 out of what appeared in the 9.5 and 9.6. And also, what new features we are waiting for new releases of PostgreSQL.

Often it is enforced by the customer or even by law, to document the circumstances of the production of parts or lots. This talk is about the automated generation and storage of the corresponding administrative data and their correlation to measurements. Administrative data are entities like suppliers batch, article, serial number or production date. They are often to be exchanged with ERP systems.

1. Why is it time to consider switching to PostgreSQL?
2. 1С and PostgreSQL interoperability.
3. Why should we start with Windows?
4. The first experience of migrating big enough 1С databases to Postgres.
5. Large-scale projects on high-load systems, in figures.

Most of the main Skype services use a database platform based on PostgreSQL and other open-source technologies, such as Skytools, plProxy, pgBouncer, etc. This platform consists of several hundreds of servers with thousands of databases, which process hundreds of thousands of transactions per second. At the same time, the platform architecture allows its users (applications and their developers) to work with "logical" databases, without any worries about their real “physical” structure.

Our Skype Database Platform team is responsible for the database platform infrastructure. We develop automation systems for various processes that help us ensure service reliability and facilitate development, testing, and deployment of code. In this presentation, I will outline the database platform architecture, review its main components, and tell you about the methods we use in our every-day work to ensure high availability, scalability, replication, fault tolerance, and more.

PostGIS enables spatial processing in Postgres. Usually PostGIS is used for planar euclidean calculations, but real world adds time, ambiguity and uncertainty. We'll have a look on making a thousand of cars move smoothly in real time on OpenStreetMap base map.

VIDEO

Practical experience of carrying out import substitution with using PostgreSQL in government information system including not only the free software, but also the Russian hardware (Elbrus servers and other).

VIDEO

Do you test your PostgreSQL releases prior to Production in a dedicated test environment? Are you sure that your test environment (shortly Test) is equal to Production and in an appropriate state?

In Skype we were facing multiple challenges associated with database testing:
- Simplifying complex Production architecture of thousands of PostgreSQL instances, interconnected with RPCs and replications, infrastructure servers and external DB scripts, into their Test counterparts.
- Constantly growing hardware requirements, insufficient cleanup of data generated in Test.
- Differences between Test and Production were appearing and accumulating. Recognizing and fixing them required lots of effort.

Real-word data often need cryptographycal protection. The presentation describes typical problems that can be solved using cryptographycal methods and the right ways to use cryptography with RDBMS. The newest solutions suggested for usage with PostgreSQL are described.

VIDEO

This talk is about migrating an electronic document management system from MS SQL to PostgreSQL 9.5 or higher as part of the import phaseout initiative. We will touch upon architecture specifics, as well as describe the problems we encountered when migrating T-SQL code to pgsql, and how we resolved them.

Learn more at https://pgconf.ru/news/94168

VIDEO

The new ISO/IEC 9075-2:2016 standard specifies JSON data and operations syntax and semantics specifics for SQL. This talk overviews the requirements of this standard, and focuses on the differences between them and the actual implementation of JSON/JSONB in PostgreSQL. Special attention will be paid to JSON Path (XPath analogue), SQL/JSON functions and our plans to make Postgres compliant with this standard.

VIDEO

Overview of the architecture of Greenplum MPP (Massively Parallel Processing) database. Explain the internals of GPDB. Show how to configure and setup GPDB. How to distribute data effectively for MPP

VIDEO

Query optimization is an important problem, which solution has a great influence on DBMS performance, especially for complex queries. In this talk we consider PostgreSQL query optimizer and specifically cardinality estimation problem for correlated clauses, which is one of the most well-known drawbacks of query optimizers in general. In the talk we propose our solution for this problem which involves machine learning methods and is available for PostgreSQL 9.6 as an extension with a patch. We discuss the experimental evaluation, advantages, disadvantages, and fields of application of the proposed approach as well.

VIDEO

A summary of what the Postgres Professional company has achieved in its two-year history:

• Our achievements in PostgreSQL development.
• What is the Russian Postgres Pro DBMS, and how is it related to PostgreSQL?
• What is Postgres Pro Enterprise, and why Enterprise?
• What about trainings and certification?

VIDEO

• why we build PostgreSQL from sources?
• choice build options
• dependencies
• creation system environment
• how to customise Linux for work with PostgreSQL
• extra soft for PostgreSQL DBA

VIDEO

1) About 2018 FIFA World Cup and tasks set; 2) Tools of the department of modeling and analytics; 3) Passenger traffic analysis in Koltsovo airport and on railway transport (long distance and suburb trains); 4) Population and employment analysis; 5) Routes design for client groups with automated metainformation display and assembling of the final reports; 6) Deployment of a map server to give access for the costumer to the client group routes

VIDEO

VIDEO

We will discuss some tricks and patterns while developing web-application architecture based on ORM technology with Postgresql functions implementation with Python Django as an example. Also we will consider some ways of building the business logic inside a database with keeping Django framework opportunities and implementation of it's built-in admin panel.

VIDEO

It often happens that you already have PosgtreSQL in production, but you don’t have a DBA. To demystify the PostgreSQL database, I’ll tell you how to troubleshoot various problems while working with PosgtreSQL. We will try to understand how to answer "routine" questions of a typical system administrator: - Is everything OK with the database? - What consumes DB server resources? - What to optimize first to reduce resource consumption?

VIDEO

Database systems are increasing in size and so is the need to process huge amounts of data in real time. As commercial database vendors are bragging about their capabilities we decided to push PostgreSQL to the next level and exceed 1 billion rows per second to show what we can do with Open Source. To those who need even more: 1 billion rows is by far not the limit - a lot more is possible. Watch and see how we did it.

VIDEO

I love complex knowledge domains, strong typing in applications, and 3NF, but I hate ORM. That is why I’ve been actively storing serialized structures in JSON fields (even before the JSON type was introduced). In this talk, I will tell you about some specifics of storing complex structures within DBMS fields, what problems can arise, and how to cushion the blow.

VIDEO

This report overviews some ideas and implementations to speedup different parts of generalized search trees (GiST): 1. Intrapage indexing 2. Fractal tree technology 3. Modern algorithms for spatial indexing (RR*-tree) 4.. Possible advancements of GiST API

VIDEO

Currently, to execute SQL queries PostgreSQL uses interpreter, which implements Volcano-style iteration model. At the same time it’s possible to get significant speedup by dynamically JIT-compiling query “on-the-fly”. In this case it’s possible to generate code that is specialized for given SQL query, and perform compiler optimizations using the information about table structure and data types that is already known at run time. This approach is especially important for complex queries, which performance is CPU-bound.

Using PostgreSQL since 2003, we went all the way from a database of a couple of GB to a cluster of more than 5TB. At the moment, we have more than 700 tables and about 1500 stored procedures. We are ready to share with you the following: - Problems encountered at different development stages and how we resolved them. - Best practices in database administration. - Our own extension to work with several closely related databases. - Best known methods and tools that enable our several teams to work together without interference. - How we set up test equipment of different types. And, of course, we'll talk about optimization, and how we identify bottlenecks and high-load use cases.

VIDEO

A short talk about collecting data and monitoring database workload in Avito. Exporting metrics from stored procedures to Graphite. Collecting and visualizing pg_stat* metrics in Grafana. Case studies.

VIDEO

Citus allows you to distribute postgres tables across many servers. It extends postgres to transparently delegate or parallelise work across a set of worker nodes, enabling you to scale out the CPU and memory available for queries.

One year ago, we began a long journey to allow Citus to scale out another dimension: write throughput. With writes being routed through a single postgres node, write throughput in Citus was ultimately bottlenecked on the CPUs of a single node. Citus MX is a new edition of Citus which allows distributed tables to be used from from any of the nodes, enabling NoSQL-like write-scalability.

Dalibo team produces open source tools for PostgreSQL among other things for many years now (see http://dalibo.github.io/).

This time I'll present temBoard, a new tool to remotely control PostgreSQL databases. The project is visible at https://github.com/dalibo/temboard

It's about monitoring, supervision, remote settings and actions... and many more features to come. The main goal is to provide a comprehensive console for PostgreSQL. It's needed by production DBAs, to daily achieve their tasks. We heard Dalibo's customers, as many of them wrote the specs :-)

Input-output (IO) performance issues have been on DBAs’ agenda since the beginning of databases. The volume of data grows rapidly and time is of an essence when one needs to get necessary data fast from the disk and, more importantly, to the disk.

For most databases it is relatively easy to find checklist of recommended Linux settings to maximize IO throughput and, in most cases, this checklist is indeed good enough. It is however essential always to understand how the optimisation of those settings actually works, especially, if you run into corner cases.

Experience we've got after 5 years of developing, deploying and improving BI system http://colibri365.ru used in government. I would talk about government IT reality and our way over it. Postgres performance improvements, using of latest features, overwriting of user generated queries to help query optimizer and other tweaks and hacks to tackle limited hardware problems. These lead us to number of computer science papers and (now committed) patches to Postgres (see Andrey Borodin talks for details).

VIDEO

1. What is an autonomous transaction?
2. An overview of autonomous transactions in "big" DBMS: Oracle.
3. Autonomous transaction logic in Postgres Pro.
4. An overview of emulation methods for autonomous transactions in PostgreSQL.
5. Comparing performance of the built-in Postgres Pro autonomous transaction mechanism and PostgreSQL emulation methods.

VIDEO

Sets are apparently a useful data type for many kinds of applications. While PostgreSQL offers no built-in set data type, sets may be emulated to some degree with its built-in array and JSONB data types. Also, acceleration of respective containment (subset) queries is readily available as a built-in feature of the GIN index type.

Starting with the above, we will then explore the performance gains enabled by custom set data types, and especially by customisation code in C ("operator classes") for the GIN and GiST index types.

< Query failed: ERROR: deadlock detected
< DETAIL: Process 17371 waits for ShareLock on transaction 102733872; Blocked by process.
< Process 10414 waits for ShareLock on transaction 102733874; Blocked by process 17371.

Such "unpleasant" messages from the server can seriously puzzle the developer. When working with locks, in particular, with transactions in general, it is necessary to take into account the features of the implementation of client libraries, which can cause the above exception.

In the short talk, the mechanics of the interaction of locks will be explained, main attention being paid to causes of deadlocks. References to the relevant documentation pages will be given. A technique of "bypassing" this problem of concurrent data access will be described and illustrated with some generalized examples from practice are shown. The talk will be interesting to database developers and administrators as well as the client-side application developers.

When working with a complex business logic, you often have to implement a workflow - a sequence of several processing steps, with each step implementing a separate part of the business logic. This is usually done with specialized queues, but if there are high reliability demands, it makes sense to do everything on PostgreSQL.

I will describe the tasks that require a workflow implementation, offer a solution, compare it with other options, and tell you about the implementation traps and pitfalls.

"1C:Enterprise" is the most popular in Russia/Eastern Europe business applications development framework supports PostgreSQL for 10+ years. We'll review some cases and pecularities of 1C and PostgreSQL tandem.

In the data persistence layer, using PostgreSQL from the very start of development, we went all the way from a small cluster on a virtual machine to a multi-host system that provides near real-time processing of mixed OLTP/OLAP load. In this talk, I’m going to tell you about the main development stages of our analytical solution at the application and infrastructure levels, and describe the specifics of using PG that we encountered.

VIDEO

VIDEO

MoySklad online service includes important component - semiautomatic database conversion framework. Its history and supported conversions are subject of this speech.

VIDEO

ZSON is a PostgreSQL extension for transparent JSONB compression. Compression is based on a shared dictionary of strings most frequently used in specific JSONB documents (not only keys, but also values, array elements, etc). In some cases ZSON can save half of your disk space and give you about 10% more TPS.

VIDEO

Alexey will present technical details and share hands-on experience of extreme data normalization application for data infrastructure with exceptional parameters design and development. Extreme normalization-based data infrastructures has the following competitive advantages in comparison with market leaders: - Real-time data processing for 10 PB of data and more - 2-6 times better overall performance - 100% data consistency through total data landscape - Almost linear scalability - 4-10 lower cost of ownership - etc The abovementioned approach has been successfully utilized out of Russian market in telecommunication, retail, fin-tech, manufacturing (Industry 4.0, industrial IoT), and government institutions.

VIDEO