
Как обфусцировать базу в Postgres для задач нагрузочного тестирования веб-приложений
Postgres is a well-known database for high load web applications. Such apps require stress/load testing itself to run properly in production. Besides obvious difficulties in preparation a test environment identical to production, generating proper traffic there is another one issue - database preparation for the test environment. And it seems it is not good to use the database from production in the testing environment in the era of personal data protection (GDPR, HIPAA). Data obfuscation is the rescue.
There are few instruments for data obfuscation in Postgres. During this session, I will tell you which of them we've selected and why what type of issues we faced, and if our solution was successful. You will know if it is possible to get an identical response on the test database without real data from production, we will observe some restriction on obfuscation, I'll present our utility which simplify things.
Видео
Другие доклады
-
Dmitry Ursegov Postgres Professional
Шардман - естественный подход к шардингу в PostgreSQL
The amount of data that is handled today by Enterprises and Web companies is constantly growing. At the same time, it becomes increasingly difficult to have and synchronize several copies of data in different systems. As a result there is a demand to work with large amounts of data directly in a transactional DBMS. This requirement is often imposed by the logic of applications that need real-time results. In this talk we will consider what a universal distributed transactional DBMS can be. We will analyze such aspects as the types of load and their prioritization, dynamic resource allocation and the level of consistency. What tools in PostgreSQL can be used to build such system, what we have already done and what is still missing.
-
Daniele Varrazzo Codice Lieve
psycopg3: как Питон полюбил Постгрес
Python is today one of the most used programming languages in the world: simple to learn and to use and ready to interface to any known service and protocol. psycopg2 is the most used PostgreSQL driver for Python: it offers good performance and makes the communication between the language and the database as smooth as possible.
Python has evolved enormously in the past years and its first-class support for async programming is changing the way new programs are written. PostgreSQL has evolved too: a new generation of the driver is needed to make the most of all the features it has to offer.
psycopg3 is the new generation of the most used Python-PostgreSQL adapter: it offers a familiar interface and smooth upgrade path, but behind the scenes it is engineered to obtain the best performance from the database and the language: async programming, prepared statements, binary parameters.
psycopg3 is also experimenting with innovative JSONB support and query pipelining! Come and discover the forefront of the research between your most loved language and database!
-
Christopher Travers DeliveryHero SE
Когда всё идёт не так: как надо и как не стоит реагировать на инциденты при поддержке СУБД
Once at Adjust we faced a problem of impending xid wraparound in a very central database due to a long-stalled autovacuum run. Because we spotted warnings 5 hours before impending disaster we were able to minimize the customer impact. Come learn how we used the time to prepare, and what lessons this has for others facing unusual problems in large databases.
-
Nikolay Samokhvalov Nombox LLC
Автоматическое тестирование изменений БД (DDL, DML)
In a heavily-loaded project, any change implies non-zero risks of downtime or performance degradation. We constantly see how system complexity, number of database nodes, deployments per week constantly grow. How grows the level of automatization of various activities in CI/CD pipelines, containers, Kubernetes.
Meanwhile, when we look at the topic of testing of database changes–from trivial index creation to complex, almost "surgical" operations like converting int4 PK to int8 one in a multi-terabyte database under load–here we observe an obvious lag in technology and methodology development. In the best case, the changes are verified and approved visually, and here it all depends on the level of experience and tiredness of the reviewer.
In this talk, we will look at how we (Postgres.ai) solve this problem using our solution, Database Lab:
- instant provisioning of independent thin clones of multi-terabyte databases, ready to be used for testing,
- integration with existing CI/CD solutions and workflow,
- collection of metrics that are the most useful for decision support on whether or not each database migration has to be approved (and even fully automated rejection of the most dangerous actions).