Boxes intermittently boot unprovisioned: fetch-ec2-metadata silently fails, amazon-init sees no user-data

[defangdevs]

Three of five deploy-test runs on master today failed the ipv4-full smoke test with 443 never answering (runs 29503622136, 29505653710, and the SSM-enabled dispatch 29507690672). The stack reaches CREATE_COMPLETE every time; the box just never serves.

The enable_ssm=true diagnostics (shipped in PR 79 for exactly this) finally showed why:

systemctl --failed            → 0 loaded units listed
Unit caddy.service could not be found.
amazon-init: /nix/store/.../amazon-init-start: line 17: /etc/ec2-metadata/user-data: No such file or directory
amazon-init: no user data is available

The instance is a vanilla NixOS AMI — the first-boot nixos-rebuild never ran, so no caddy/ttyd/anything. Root cause is in the stock AMI's fetch-ec2-metadata.service (ec2-metadata-fetcher.sh):

1. The IMDSv2 token fetch uses --connect-timeout 1 with a handful of 1s retries (~10-15s total window). If IMDS is slow to respond at boot, it logs failed to fetch an IMDSv2 token. and continues with an empty token.
2. The subsequent user-data fetch 401s and is swallowed by || true — no file written, unit exits 0 (hence zero failed units).
3. amazon-init finds no file, prints "no user data is available", also exits 0.

Net effect: a transient IMDS hiccup at first boot silently yields a permanently unprovisioned box. This is not CI-only — 1-click users hit the same dice roll, and for them it looks like a dead WebURL with a green CloudFormation stack. Upstream awareness: nixpkgs issue 449166 (open) is adjacent; nixpkgs PR/issue 447266 added the retries but kept the fail-open behavior.

Also: the deploy-test step "Assert amazon-init provisioned the box (serial console)" passed on this unprovisioned box — it greps for Finished ... amazon-init, and a no-user-data boot finishes "successfully" having done nothing.

Proposed fixes, in order of leverage:

1. CI fail-fast (cheap, done first): fetch-ec2-metadata logs with StandardOutput=journal+console, so failed to fetch an IMDSv2 token. IS visible in get-console-output. Make the serial-console assertion fail immediately on that marker (and on no user data is available if it proves console-visible) instead of burning 10 minutes of curl 000 and misattributing the failure.
2. Product-level visibility: add a CreationPolicy/WaitCondition to the template, signaled (plain curl to the presigned URL, no cfn tools needed) at the end of our user-data activation. An unprovisioned box then turns into a visible CREATE_FAILED instead of a silently dead URL. Design call: rollback vs. keep-for-debug, timeout length.
3. Self-healing: our user-data can't fix this (it's precisely what never runs). Real options are a custom AMI, or fixing upstream (make the fetcher fail the unit + Restart=on-failure, or make amazon-init require the user-data file when the instance has user data). Worth an upstream issue/PR referencing this evidence.

[defangdevs]

New data point: the post-PR-85 master run (29513819781) failed the ipv4-full smoke test the same way (10 min of curl 000), but with a clean Finished amazon-init and no failed to fetch an IMDSv2 token marker on the console. So that run is either:

• the console-invisible variant — token OK, but the single-shot get_imds ... || true user-data GET failed (no retry, --fail --silent, error swallowed), or
• a genuinely provisioned box that's dead for some other reason (ACME, caddy, EIP path).

Push runs had no SSM, so the evidence was destroyed with the stack. PR 86 makes the ipv4-full leg always run with EnableSsm=true and extends the failure dump with the fetch-ec2-metadata journal, /etc/ec2-metadata/ listing, and ss -ltn — the next failure will distinguish these two.

Score for 2026-07-16 so far: 4 failures / 6 runs, all ipv4-full smoke, zero CREATE_FAILED.

[defangdevs]

Another occurrence, 2026-07-17: master run 29583478064 (job ipv4-full), FirstBootDone: WaitCondition timed out. Received 0 conditions when expecting 1 — https://github.com/defangdevs/agent-box/actions/runs/29583478064. The ipv6-outputs job in the same run provisioned fine, so it's the usual intermittent case.

No on-box evidence captured this time: the "Diagnose auth state via SSM" step printed "No InstanceId; skipping SSM diagnostics" — it reads the InstanceId from stack OUTPUTS, which don't exist when create fails. It could instead resolve the instance from the failed stack's resources (aws cloudformation describe-stack-resources --stack-name ... --logical-resource-id EC2Instance → PhysicalResourceId), which works in CREATE_FAILED before the destroy step tears it down. As it stands, the diagnostic step is blind in exactly the failure mode it was added for.