PostgreSQL and MQTT as IoT data processing solution

MQTT is an effective data exchange protocol for IOT devices. Based on own modified EMQTT plug-in, the IoT project architecture uses PostgreSQL as a central processing and storing system for data coming from sensors in real time. The report will provide an example of the IoT hardware and software platform solution, based up on the MQTT protocol, where PostgreSQL is responsible for key functionality, providing processing, collecting and storage of data from a distributed network of IoT devices.

Cases for monitoring & troubleshooting at highload PosgreSQL DB

A database is one of the key components in any information system, requiring the monitoring of multiple metrics. The talk highlights examples and approaches of monitoring and analysis of PostgreSQL performance that allow to minimize the load on the database server from the monitoring and data collection system for the subsequent analysis of problem situations.

• Quantum effects or as an observer affect the observed system
• Features of collecting metrics while monitoring the database with Zabbix
• Data collection for analytics and visualization PostgreSQL queries with rsyslog + kafka + clickhouse + grafana
• Operational Analysis Tools for DB loglile

PostgreSQL SaaS in the Rambler&Co

Rambler & Co is a lot of publications, services and projects. Appear new and grow existing. This environment requires a reliable, fault-tolerant, scalable, automated system.

I'll tell you about the structure of our PostgreSQL SaaS, what tools and technologies we use. Quorum of 3 Data Centers. A single entry point for clients based on dynamic routing. Emergency switching of the primary server. Transparent scaling for reading. Create a replica without load on the cluster. Transparent transfer of PostgreSQL cluster to other servers. Update dev environment from prod for development. Backup with compression and the use of multiple CPUs on the side of the database, the restoration of one database from basebackup. Monitoring of sql queries.

Jsonb flexible indexing. Parameterized access methods operator classes.

Jsonb is a popular data type in PostgreSQL, it provides the web developers an ability to work with ubiquitous json inside the database and use all the power of proven relational database. Fast querying of jsonb data is a challenge for database and PostgreSQL provides several options for indexing jsonb. We present the new way of efficient indexing of jsonb, based on improvement of indexing infrastructure.

It's known, that json is a greedy data type, it may contains many auxiliary data not interesting for searching and that affects the size of index. Partial index will not helps, since it filters the rows before indexing, while we are interested in extracting of parts of jsonb. Functional indexes on specific keys could introduce too big overhead. We present an improvement of indexing infrastructure, which allows to control the index behaviour by passing parameters to operator class at index creation. For example, to index a user-defined subset of jsonb it is possible to pass to operator class the powerful path expression (either jsonpath of upcoming sql/json or jspath from jsquery extension), which can be used to extract the parts of jsonb tree. That makes index more effective and reduces the overhead of its maintaining.

Another use of parameterized operator classes is to allow a user to specify parameters instead of hard coding them, for example, the GiST signature size is currently hard coded inside the implementations of several opclasses (tsvector, hstore, intarray, pg_trgm, ltree), while it is natural to use different signature length for different data to have optimal size of index and its performance.

Full text search on parts of document can be improved by passing labels to the operator class and letting him index only specified parts of document, that allow to avoid currently used recheck of the rows returned by the index.