Since PR 110, the first-boot gate turns the IMDS brick (issue 106, upstream NixOS/nixpkgs#542874) into CREATE_FAILED + rollback. First in-CI catch: master run 29571322617 — the ipv6-outputs leg failed honestly (previously it reported green over a dead box) while the ipv4-full leg passed the full smoke test in the same region at the same time, i.e. the race is intermittent, not a regional outage.
Until the upstream fetch-ec2-metadata fix lands, the race will keep flaking deploy-test runs. Proposal: in the Create stack step, on a CREATE_FAILED whose stack events show FirstBootDone ... WaitCondition timed out, delete the rolled-back stack and retry the create exactly once (a second consecutive brick should stay a failure — it likely isn't the race). Keep the retry OFF for dispatched runs when destroy=false (debugging wants the artifact, though rollback already tears it down).
Not a fix for end users clicking Launch — they see a clean CREATE_FAILED and relaunch — but it keeps master CI signal usable while upstream is broken.
Since PR 110, the first-boot gate turns the IMDS brick (issue 106, upstream NixOS/nixpkgs#542874) into CREATE_FAILED + rollback. First in-CI catch: master run 29571322617 — the ipv6-outputs leg failed honestly (previously it reported green over a dead box) while the ipv4-full leg passed the full smoke test in the same region at the same time, i.e. the race is intermittent, not a regional outage.
Until the upstream fetch-ec2-metadata fix lands, the race will keep flaking deploy-test runs. Proposal: in the Create stack step, on a CREATE_FAILED whose stack events show
FirstBootDone ... WaitCondition timed out, delete the rolled-back stack and retry the create exactly once (a second consecutive brick should stay a failure — it likely isn't the race). Keep the retry OFF for dispatched runs whendestroy=false(debugging wants the artifact, though rollback already tears it down).Not a fix for end users clicking Launch — they see a clean CREATE_FAILED and relaunch — but it keeps master CI signal usable while upstream is broken.