Give the backup loop its clock, its alarm, and its tripwire - #589
Merged
Conversation
The backup mechanism shipped proven but inert: the conf file itself admitted nothing schedules backups (so nothing expires them), nothing alerts on a failing archive, and nothing runs verify. Each admission now has a mechanism, and the one decision with a deadline has a guard. infra/backup/systemd/ ships four timers and their services as inert artifacts: weekly full and daily differential (retention rides the backup command, so the backup timers are also the expiry mechanism), hourly check (a real WAL segment pushed through the archive and confirmed, which is the broken-archive alert before the failure mode of filling the volume), and weekly verify (every stored checksum read back, so bit rot surfaces on a timer rather than at the next restore). Every unit routes failure through one cora-backup-failure@ template, shipped writing a high-priority journal line and existing to be the single place a deployment attaches its pager. Whether the operating account may install any of this is HOST-5 on the 2-BM questions page; the artifacts are ready either way, and the docs carry the docker alternative. The encryption deadline becomes unforgettable by mechanism: test_pgbackrest_conf.py fails CI on any repo1-type other than posix that lacks a repo1-cipher-type, because encryption is fixed at stanza creation and re-pointing the repository without deciding it would otherwise merge silently. The same file pins retention-as-count (the time type silently changes WAL archive retention too), refuses an explicit archive-check=n (the setting whose removal makes backups unrestorable while every command reports success), and the stanza credentials whose absence reads like a broken installation. The restore drill gains a verify phase at the moment it means the most: after the WAL tail is pushed and before the source is destroyed, certifying the soundness of the only copy about to exist. Exercised live: the full drill passed with the new phase on this change. The deferred background-scheduler row records its first firing: the backup job needed to outlive a process and was answered at the host layer, not in the application; the row stays open for the first app-internal job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Coverage report
This PR does not seem to contain any modification to coverable code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the operational gaps the backup mechanism (#582) honestly documented about itself: nothing scheduled backups (so nothing expired them), nothing alerted on a failing archive, nothing ran
verify, and the encryption deadline was a sentence rather than a mechanism.What ships
infra/backup/systemd/— four timers and services as inert artifacts, ready for the host HOST-5 describes:cora-backup-fullcora-backup-diffcora-backup-checkcora-backup-verifyEvery unit routes failure through one
cora-backup-failure@.servicehook (ships as a high-priority journal line; the single place a deployment attaches its pager). All timers arePersistent=trueso a host down over the window catches up instead of skipping a week.The encryption tripwire —
tests/unit/deployments/test_pgbackrest_conf.pyfails CI on anyrepo1-typeother thanposixwithout arepo1-cipher-type. Encryption is fixed at stanza creation; re-pointing the repository without deciding it can no longer merge silently. The same file pins retention-as-count (thetimetype silently changes WAL archive retention too), refuses an explicitarchive-check=n, and the stanza credentials.The drill gains a
verifyphase at the moment it means most: after the WAL tail lands and before the source is destroyed, certifying the soundness of the only copy about to exist. Exercised live: the full drill passed with the new phase on this change, restore assertions included.Docs: the three now-closed "Limits, stated plainly" bullets become mechanisms with the honest remainder (installation is HOST-5's question; a docker deployment runs the same four commands via the compose tools profile), and the deferred background-scheduler row records its first trigger firing, answered at the host layer with the row staying open for app-internal jobs.
🤖 Generated with Claude Code