fix: retry startup auto-run while the database worker is unavailable - #1
Merged
Conversation
With auto: true the startup migration run was attempted exactly once. When miiigrate registered before the database worker — the common case under docker compose or an engine-managed boot, where registration order is not guaranteed — the run failed with DATABASE_WORKER_UNAVAILABLE and was never retried, silently leaving migrations unapplied until a manual migrate::up. The auto run now retries on DATABASE_WORKER_UNAVAILABLE with capped exponential backoff (1s doubling to 10s, 2-minute budget). Any other error, and budget exhaustion, remain loud and terminal for the auto run only: the worker stays up so migrate::status and migrate::up are available for diagnosis and manual recovery.
… connection errors Two more sources of apparent errors during a normal cold boot, where the engine and its workers come up in arbitrary order: - configuration::register/get gave up after 3 attempts (~16 s) and crashed the worker with a startup error, leaving the restart policy to try again. Transient failures (timeout, not connected, function not registered yet) are now retried with capped backoff under the same 2-minute budget as the auto run; real configuration-worker errors still fail fast. - iii-helpers' OTel side channel logs every pre-connection attempt at ERROR (hardcoded in the helper, no config knob) while iii-sdk already reports the identical condition as a WARN retry. The default tracing filter now turns that module off; setting RUST_LOG replaces the filter entirely.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With
auto: truethe startup migration run was attempted exactly once. When miiigrate registered before thedatabaseworker — the common case under docker compose or an engine-managed boot, where registration order is not guaranteed — the run failed withDATABASE_WORKER_UNAVAILABLEand was never retried, silently leaving migrations unapplied until a manualmigrate::up.More generally, a normal cold boot produced apparent errors: the
configuration::register/getcalls gave up after ~16 s and crashed the worker (leaving the restart policy to recover), andiii-helpers' OTel side channel logged every pre-connection attempt at ERROR.Fix
DATABASE_WORKER_UNAVAILABLEwith capped exponential backoff (1 s doubling up to 10 s, 2-minute budget). Any other error — and budget exhaustion — remains loud and terminal for the auto run only: the worker stays up somigrate::statusandmigrate::upare available for diagnosis and manual recovery, unchanged from before.iii_helpers::observability::telemetry::connection, which logs every pre-connection attempt at ERROR (hardcoded in the helper) while iii-sdk already reports the identical condition as a WARN retry. SettingRUST_LOGreplaces the filter entirely.Docs updated:
docs/configuration.md(autorow) anddocs/changelog.md(Unreleased).Validation
cargo build,cargo test(47 tests),cargo clippy— clean.