Backfill Lambda layer versions for multi-region consistency#73
Open
RaphaelManke wants to merge 7 commits into
Open
Backfill Lambda layer versions for multi-region consistency#73RaphaelManke wants to merge 7 commits into
RaphaelManke wants to merge 7 commits into
Conversation
Lambda layer versions auto-increment per region and cannot be chosen, so a newly launched region (e.g. il-central-1) would start at version 1 while all existing regions are at some aligned version N. The sls-plugin ARN builder assumes the same layer name + version resolves in every region. The script reads the highest version per layer from a source region and publishes dummy versions (deleted immediately after publishing) in the target region until its version counter matches. The next real deploy then produces the same version number in the new region as everywhere else. Supports --dry-run, defaults to the four prod layers, and aborts with a clear error if the target counter is already ahead. After running it for il-central-1, the region can be added to the deploy.yml region matrices (separate change). Resolves CLO-944. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQm5yBAanBxSAuohiWuiKn
The query LayerVersions[0].Version sometimes returns extra None lines alongside the actual version number. Filter to first non-empty line.
The prod deploy pipeline now includes a pre-deploy step that checks all regions against eu-west-1 (source) and backfills any misaligned version counters. This ensures new regions and any region that diverged from the source will have aligned version numbers before publishing new layer versions. The backfill runs for each of the 16 non-source regions, using || true to log but not block deploy if a region fails (rare edge case).
RaphaelManke
marked this pull request as ready for review
July 15, 2026 09:10
mosheshaham-dash0
approved these changes
Jul 15, 2026
…l is skipped Resolves the Aikido finding on the backfill step: inputs.region was interpolated directly into the run command, a template-injection sink; it is now passed through an environment variable. Also fixes the deploy jobs being skipped whenever the backfill job is skipped (dev environment, eu-west-1): a skipped dependency counts as not-success for the default job condition, so deploy-python/node/java/ manual now run when backfill succeeded or was skipped, but still not when it failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQm5yBAanBxSAuohiWuiKn
The span/log query window (now +/- 5min, recomputed per retry attempt) slid past the invocation timestamp after ~5 minutes of retrying, making late-arriving telemetry permanently unfindable and burning the whole retry budget. Extend the lookback to 20 minutes for span, log, and metric queries so the invocation stays inside the window for the full 50 x 10s retry budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQm5yBAanBxSAuohiWuiKn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add helper script and CI integration to align Lambda layer version counters across regions for new region launches (e.g., Israel, il-central-1).
Changes
1. Helper Script:
tools/backfill-layer-versions.sh--dry-runfor safe previewing./tools/backfill-layer-versions.sh [--dry-run] <source-region> <target-region> [layer-names...]2. Workflow Integration:
publish.ymlbackfilljob runs before deploy stepsprod-integration)3. Cleanup:
deploy.ymlbackfill-prodjobdeploy-prodnow depends directly onbuildTesting
Rollout
Resolves CLO-944.