Skip to content

fix: retry startup auto-run while the database worker is unavailable - #1

Merged
heryr merged 3 commits into
mainfrom
fix/auto-run-retry-database-unavailable
Jul 21, 2026
Merged

fix: retry startup auto-run while the database worker is unavailable#1
heryr merged 3 commits into
mainfrom
fix/auto-run-retry-database-unavailable

Conversation

@heryr

@heryr heryr commented Jul 21, 2026

Copy link
Copy Markdown
Member

Problem

With auto: true the startup migration run was attempted exactly once. When miiigrate registered before the database worker — the common case under docker compose or an engine-managed boot, where registration order is not guaranteed — the run failed with DATABASE_WORKER_UNAVAILABLE and was never retried, silently leaving migrations unapplied until a manual migrate::up.

More generally, a normal cold boot produced apparent errors: the configuration::register/get calls gave up after ~16 s and crashed the worker (leaving the restart policy to recover), and iii-helpers' OTel side channel logged every pre-connection attempt at ERROR.

Fix

  • Auto-run retry — the startup run retries on DATABASE_WORKER_UNAVAILABLE with capped exponential backoff (1 s doubling up to 10 s, 2-minute budget). Any other error — and budget exhaustion — remains loud and terminal for the auto run only: the worker stays up so migrate::status and migrate::up are available for diagnosis and manual recovery, unchanged from before.
  • Configuration RPC retry — transient failures (timeout, not connected, function not registered yet) are retried with capped backoff under the same 2-minute budget instead of crashing the worker; real errors from the configuration worker still fail fast.
  • No duplicate ERROR noise — the default tracing filter silences iii_helpers::observability::telemetry::connection, which logs every pre-connection attempt at ERROR (hardcoded in the helper) while iii-sdk already reports the identical condition as a WARN retry. Setting RUST_LOG replaces the filter entirely.

Docs updated: docs/configuration.md (auto row) and docs/changelog.md (Unreleased).

Validation

  • cargo build, cargo test (47 tests), cargo clippy — clean.
  • Reproduced end to end in a dockerized stack (engine 0.21.6 + Postgres 16 + database worker, miiigrate built from this branch), cold start with no ordering guarantee. Zero ERROR lines; the full boot log is INFO plus three legitimate transient WARNs:
INFO  miiigrate: starting name="miiigrate" …
WARN  iii_sdk::iii: failed to connect; retrying error=IO error: Connection refused (os error 111)
INFO  iii_sdk::iii: iii connected address=ws://iii:49134
INFO  miiigrate: configuration loaded db=primary dir=./migrations auto=true
WARN  miiigrate: database worker not ready; retrying auto migration run … retry_in_s=1
WARN  miiigrate: database worker not ready; retrying auto migration run … retry_in_s=2
INFO  miiigrate::handlers::up: applied migration=20260101120000_create_users.sql
INFO  miiigrate: auto migration run complete …
INFO  miiigrate: miiigrate worker registered 4 functions, waiting for invocations

heryr added 3 commits July 21, 2026 15:32
With auto: true the startup migration run was attempted exactly once. When
miiigrate registered before the database worker — the common case under
docker compose or an engine-managed boot, where registration order is not
guaranteed — the run failed with DATABASE_WORKER_UNAVAILABLE and was never
retried, silently leaving migrations unapplied until a manual migrate::up.

The auto run now retries on DATABASE_WORKER_UNAVAILABLE with capped
exponential backoff (1s doubling to 10s, 2-minute budget). Any other error,
and budget exhaustion, remain loud and terminal for the auto run only: the
worker stays up so migrate::status and migrate::up are available for
diagnosis and manual recovery.
… connection errors

Two more sources of apparent errors during a normal cold boot, where the
engine and its workers come up in arbitrary order:

- configuration::register/get gave up after 3 attempts (~16 s) and crashed
  the worker with a startup error, leaving the restart policy to try again.
  Transient failures (timeout, not connected, function not registered yet)
  are now retried with capped backoff under the same 2-minute budget as the
  auto run; real configuration-worker errors still fail fast.

- iii-helpers' OTel side channel logs every pre-connection attempt at ERROR
  (hardcoded in the helper, no config knob) while iii-sdk already reports
  the identical condition as a WARN retry. The default tracing filter now
  turns that module off; setting RUST_LOG replaces the filter entirely.
@heryr
heryr merged commit e02862f into main Jul 21, 2026
5 checks passed
@heryr
heryr deleted the fix/auto-run-retry-database-unavailable branch July 21, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant