Make ycmdb.shared_archive runtime-changeable and add space saver (PG18) - #91
Open
x4m wants to merge 45 commits into
Open
Make ycmdb.shared_archive runtime-changeable and add space saver (PG18)#91x4m wants to merge 45 commits into
x4m wants to merge 45 commits into
Conversation
This is squash of: 3c5eaebfcb7 deb fix 182529cdc33 Add CI e037397442c Pull request #201: bump llvm to 18 17.0 62788dae32c Disable tests with DEB_BUILD_OPTIONS af747a9ef78 Yandex build infra commit 3363cafda54 Remove stop version 22bc36aecea Lower postgresql-common and remove tzdata restriction 2b4d289397a Apply debian patch series f096dadfae6 Add debian for PostgreSQL 18 Author: Andrey Lyarskiy <aslyarskiy@yandex-team.ru> Author: EinKrebs <ein-krebs@yandex-team.ru> Author: reshke <reshke@double.cloud>
Currently we allow to cancel awaiting of syncronous commit. Some drivers cancel query after timeout. If application will retry idempotent query, it will get confirmation of written data. This can lead to split-brain in HA scenarios. To prevent it this we add synchronous_commit_cancelation setting disalowing cancelation of syncronous replication wait Version for PostgreSQL 16
Introduces 3 functions: extern bool mdb_admin_allow_bypass_owner_checks(Oid userId, Oid ownerId); extern void check_mdb_admin_is_member_of_role(Oid member, Oid role); extern bool mdb_admin_is_member_of_role(Oid member, Oid role); To check mdb admin belongship and role-to-role ownership transfer correctness. Our mdb_admin ACL model is the following: * Any roles user or/and roles can be granted with mdb_admin * mdb_admin member can tranfser ownershup of relations, namespaces and functions to other roles, if target role in neither: superuser, pg_read_server_files, pg_write_server_files nor pg_execute_server_program. * Allow mdb_admin to create LEAKPROOF functions * mdb admin sets session replication role * [MDB-16648 + MDB-17910]: Allow mdb admin to kill specific superuser queries MDB-27288: allow mdb_admin to kill autovac + tests mdb-27228: fix expected output One more fix Fix warns
MDB replication role regression tests Patch allows user with mdb_replication role to use pg_create_logical_replication_slot, pg_replication_slot_advance, pg_drop_replication_slot functions to manage logical replication slots. Also, users with mdb_admin (which is memner of pg_create_subscription) can create subscribptions. Slot names starting with MDB.* are forbidden. Add run as owner tap tests More test cases in mdb_102 Fix test Fix tests once again Never check for superuser in walsender
This commit introduces new mdb internal role mdb_superuser. Role is capaple of: GRANT/REVOKE any set of priviledges to/from any object in database. Has power of pg_database_owner in any database, including: DROP any object in database (except system catalog and stuff) Role is NOT capaple of: Create database, role, extension or alter other roles with such priviledges. Transfer ownership to /pass has_priv of roles: PG_READ_ALL_DATA PG_WRITE_ALL_DATA PG_EXECUTE_SERVER_PROGRAM PG_READ_SERVER_FILES PG_WRITE_SERVER_FILES Fix configure.ac USE_MDBLOCALES option handling Apply autoreconf stuff Set missing ok parameter ito true while acquiring mdb_superuser oid In regress tests, nobody creates mdb_superuser role, so missing ok is fine Fix spelling Applied suggestion Allow mdb_superuser to have power of pg_database_owner Allow mdb_superuser to alter objects and grant ACl to objects, owned by pg_database_owner. Also, when acl check, allow mdb_superuser use pg_database_owner role power to pass check regression test fixes
… non-superuser It is well known that some of PostgreSQL-related security issues (CVE) was related to COPY FROM/TO PROGRAM exploits for priviledge escalation or other unwanted behaviour or consequences. Thus, proper usage of this feature needed. For now, simply forbit this. Add mdb copy test
Do not use mdb_locales and mdb_newlocale when configured without them.
Added ifdef codepath build without mdb-locales feature
Add mdb locales patch, restore COPY from/to files, enable regress.
Squashed commit of the following:
commit 9f8ea4a5f42e0fd6077061bafbb428e88499896f
Author: reshke kirill <reshke@double.cloud>
Date: Wed Feb 22 09:27:53 2023 +0000
Add mdb locales patch, restore COPY from/to files, enable regress.
This commit does several things.
* Enables back COPY from/to FILE functionality, becuase it is used
by pg_regress
* Enables pg_regress tests in deb build itself.
* Add mdb locales function and checks that package build with
mdb-locales support
Add configure ac target to define USE_MDBLOCALES properly
Refactor optional setlocale, fix minor issues
Add missing SETLOCALE and NEWLOCALE calls.
They seem to be missing after 37851a8 and 4f5cef2
Co-authored-by: reshke reshke@double.cloud
Accept new startup param _pq_.service_auth_role for service auth under any user, which is niether superuser nor have some dangerous system role priv Add tap-test for mdb service role auth 👍👌😉 Fix tests after rebase contrib tests 💅️️💅️️💅️️ now works MDB-23247: debug ouput for testing purposes lowered to DEBUG5 elog level Skip caching routines for pre-startup logic (SCRAM -service auth role)
Update mdb-patched.md Update mdb-pacthes.md
Add dropdb TAP test
We learned from the field incidents, that 128Kb is not enough. This time we are going to try to increase this value.
Use fadvise in walsender remove bogus progress reporting Fix fadvise patch
There is no need to log the entire query, because it may be large and take lots of space on disk. Parameter max_log_size set the maximum length for logged query. Everything beyond that length is truncated. Value 0 disables the parameter.
Add new parameter to VACUUM command, FORCE, meaning VACUUM should terminate all backends that prevents the execution by holding conflicting lock Fix vacuum force Fix vacumm fixes for force vacuum Signed-off-by: roman khapov <r.khapov@ya.ru> (cherry picked from commit 5a312d2) Signed-off-by: roman khapov <r.khapov@ya.ru> Fix vacuumdb TAP
Also truncate query to be logged in simple query Remove unused functions in buf_internals.h
In the case of a large PGSS_TEXT_FILE, the work time of the qtext_load_file function will be quite long, and the query to the pg_stat_statements table will not be cancellable, as there is no CHECK_FOR_INTERRUPT in the function. Also, the amount of bytes read can reach 1 GB, which leads to a slow read system call that does not allow cancellation of the query. Testing the speed of sequential read using fio with different block sizes shows that there is no significant difference between 16 MB blocks and 1 GB blocks. Therefore, this patch changes the maximum read value from 1 GB to 16 MB and adds INTERRUPTS_PENDING_CONDITION() check in the read loop of qtext_load_file to make it cancellable. For now, only statement execution is cancellable (fail_on_interrupt is true only for calls from pg_stat_statements_internal) Signed-off-by: rkhapov <r.khapov@ya.ru> Reviewed-by: reshke <reshke@double.cloud>
Allow user to cancel backend which is explicitly marked as service operation, allowed to be canceled. We check `application_name` for containing MDB prefix Fix MDB-signal process and add TAP test for feature
Introduce a new archive_mode setting "shared" to prevent WAL history
loss during standby promotion in HA streaming replication setups.
In shared mode, the primary proactively sends archival status updates
to standbys via the replication protocol. The standby creates .ready
files for received WAL segments but defers marking them as .done until
the primary confirms archival. This prevents WAL from being recycled
before it's safely archived, addressing a critical gap in PITR continuity
during failover.
Key implementation details:
- Primary periodically sends last archived WAL segment via new
PqReplMsg_ArchiveStatusReport ('a') message
- Standby marks all segments <= reported segment as .done using
alphanumeric comparison on segment part (timeline-safe)
- Archiver skips during recovery in shared mode, activates on promotion
- Cascading replication: each standby coordinates with immediate upstream
- Startup check rejects archive_mode=on during recovery
This "push" design (primary sends status) is more efficient than "pull"
(standby queries per-segment), avoiding directory scans and stat() calls.
Based on Heikki Linnakangas's 2014 design and Greenplum's production
implementation, modernized for PostgreSQL 19.
Includes TAP tests covering basic synchronization, promotion,
cascading replication, and multiple standbys scenarios.
When standby receives archive status report, check if .ready files belong to ancestor timelines before the switch point and mark them as .done if already archived by primary.
When archive status reports arrive sequentially on the same timeline, directly generate expected WAL filenames and mark them as archived instead of scanning the entire archive_status directory. This optimization reduces overhead in the common case where the primary continuously archives segments. Directory scan is still used when: - Timeline changes (to handle ancestor timelines) - First report received - Non-sequential reports XLogArchiveForceDone() handles all cases internally (checking if .done exists, if .ready exists, or creating .done if neither exists), so no pre-check is needed.
XXX do we also need DROP OWNED to work this way? Probably not. Discussion: https://postgr.es/m/CALdSSPhjONb+EftRD=J1pqajkB+pjT0=tbMJs16C6q9+xT8NNg@mail.gmail.com
Reviewed-by: default avatarAndrey Borodin who is also Major PostgreSQL contributor by the way <x4mmm@yandex-team.ru>
1. Checkpoint on standby deletes WAL with .ready status. XLogArchiveCheckDone() treated archive_mode=shared like archive_mode=on during recovery, returning true unconditionally and allowing checkpoint to remove WAL segments that the primary had not yet archived. Fix: exclude shared mode from the early-return path, same as "always". 2. Walsender never sends archival status reports after archiving is restored. WalSndArchivalReport() calls pgstat_fetch_stat_archiver() whose result is cached per-session (PGSTAT_FETCH_CONSISTENCY_CACHE by default). The walsender has no transaction boundaries that would clear the cache, so last_archived_wal remained "" forever, and strcmp() suppressed all reports. Fix: call pgstat_clear_snapshot() before fetching archiver stats. Add TAP tests in 051_archive_shared_checkpoint.pl that reproduce both bugs, and extend 050_archive_shared.pl with checkpoint/restore scenarios. Reviewed-by: reshke <reshke@double.cloud>
…le (#86) For all in-core (contrib) extension, we already switch trusted options to false. For third-party extensions, simply never trust nothing. This makes API management of extensions consistent.
In MDB, not-superuser cannot install plperl. Adjust regression test for that.
This patch switches palloc to our newly preferred palloc_array and modernizes ereport calls, switching from list-style to polymorphic ereport calls. This patch also fixes whitespace/tab issues, enforcing a single style. across existing ginfuncs.c code. Inspired by Peter Eisentraut's patch in the thread. Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Japin Li <japinli@hotmail.com> Discussion: https://postgr.es/m/CALdSSPiN13n7feQcY0WCmq8jzxjwqhNrt1E=g=g6aZANyE_OoQ@mail.gmail.com
This patch provides a new version for the pageinspect contrib module, including two new functions: * gin_entrypage_items. * gin_datapage_items. These two functions can be used to examine the GIN entry tree and posting tree pages. Namely, gin_entrypage_items can be used for both leaf and non-leaf entry tree pages. gin_datapage_items is provided in pairs with already-existing gin_leafpage_items to examine the non-leaf posting tree pages. We keep the different functions here mainly because of different GIN. pages layoff. Note that fast-list pages are out of scope of this patch. Co-authored-by: Peter Eisentraut <peter@eisentraut.org> Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru> Reviewed-by: Roman Khapov <rkhapov@yandex-team.ru> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Japin Li <japinli@hotmail.com> Discussion: https://postgr.es/m/CALdSSPiN13n7feQcY0WCmq8jzxjwqhNrt1E=g=g6aZANyE_OoQ@mail.gmail.com
Suggested-by: Japin Li <japinli@hotmail.com>, Andrey Borodin <x4mmm@yandex-team.ru> Discussion: https://postgr.es/m/CALdSSPiN13n7feQcY0WCmq8jzxjwqhNrt1E=g=g6aZANyE_OoQ@mail.gmail.com
Change ycmdb.shared_archive from PGC_POSTMASTER to PGC_SIGHUP so the shared archive behavior can be toggled without a server restart. Add a new GUC ycmdb.shared_archive_space_saver (PGC_SIGHUP). When enabled on a standby it disables the shared-archive WAL retention: the walreceiver forces streamed segments to .done instead of holding them as .ready, and XLogArchiveCheckDone() lets checkpoint recycle such segments during recovery. This lets a disk-usage watcher relieve a standby that would otherwise accumulate unarchived WAL until it fills the disk and fails over (e.g. while the shared archive storage is unavailable). The trade-off is a gap in the archived WAL history; the flag has no effect on a primary. The new SharedArchiveRetentionActive() macro expresses "shared archive is active and retention is not overridden" and is used in the two retention decision points.
x4m
changed the title
reshke
force-pushed
the
MDB_18_STABLE
branch
3 times, most recently
from
July 8, 2026 06:25
ea69437 to
ae52e50
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change ycmdb.shared_archive from PGC_POSTMASTER to PGC_SIGHUP so the shared archive behavior can be toggled without a server restart.
Add a new GUC ycmdb.shared_archive_space_saver (PGC_SIGHUP). When enabled on a standby it disables the shared-archive WAL retention: the walreceiver forces streamed segments to .done instead of holding them as .ready, and XLogArchiveCheckDone() lets checkpoint recycle such segments during recovery. This lets a disk-usage watcher relieve a standby that would otherwise accumulate unarchived WAL until it fills the disk and fails over (e.g. while the shared archive storage is unavailable). The trade-off is a gap in the archived WAL history; the flag has no effect on a primary.
The new SharedArchiveRetentionActive() macro expresses "shared archive is active and retention is not overridden" and is used in the two retention decision points.