Skip to content

fix(fleet): dial the registry address so a stale ssh_config can't hide a live box#1315

Merged
muqsitnawaz merged 2 commits into
mainfrom
fleet-ssh-hardening
Jul 20, 2026
Merged

fix(fleet): dial the registry address so a stale ssh_config can't hide a live box#1315
muqsitnawaz merged 2 commits into
mainfrom
fleet-ssh-hardening

Conversation

@muqsitnawaz

Copy link
Copy Markdown
Contributor

Why

agents fleet status hangs ~60s and shows reachable devices as probe failed / ssh: connect to host <old-ip> when the user's ~/.ssh/config has a hand-written Host <name> block with a DHCP-drifted LAN IP. Root cause: the fleet probes dial the bare device name, letting ssh resolve it through ~/.ssh/config — where the stale block shadows the correct entry. The registry already stores the drift-proof Tailscale address (dnsName/IP), and the stats probe already uses it (via sshTargetFor) — which is why stats reached the exact nodes the doctor probe couldn't.

This affects any user with Tailscale + a stale ssh_config entry, not one fleet.

What

  • fleetDialTarget(device) (pure, tested) — prefer the registry Tailscale dnsName/IP; fall back to the bare name for address-less manual devices (never worse than before).
  • All three fleet probes (--version, doctor --json, fleet ping) now dial target.dialTarget instead of target.name — a stale ~/.ssh/config alias can no longer shadow the known-good address.
  • Fail fast: a device the stats probe already found reachable: false is skipped straight to an unreachable row instead of eating a 15s+30s version+doctor timeout — one dead box can't stall the matrix.

Verification

  • Live, on the real fleet, with no ~/.ssh/config change: yosemite-m1..m6 (all had drifted LAN IPs) went from
    ○ yosemite-m1  probe failed   ssh: connect to host 192.168.1.101 …
    
    to full healthy doctor rows:
    ◐ yosemite-m1  linux  idle  1/9  1.20.70  1%/6%   …
    
  • 4 new unit tests for fleetDialTarget (dnsName preference, IP fallback, no-user, address-less fallback).
  • tsc --noEmit clean; existing suite unaffected.

Follow-up (not in this PR)

agents devices sync could re-resolve + warn when a stored address goes stale (drift detection), and fleet status could stream rows as they arrive rather than buffering behind the slowest probe. Both are additive on top of this.

…ssh_config can't hide a live box

The fleet probes (version/doctor/fleet-ping) called sshExecAsync(target.name),
letting ssh resolve the bare name through ~/.ssh/config — where a hand-written
'Host <name>' block with a DHCP-drifted LAN IP silently shadows the correct
entry, times out, and (buffered behind the slowest probe) hangs fleet status
~60s while a reachable box renders as 'probe failed'. The stats probe already
dials the registry address (sshTargetFor), which is why it reached the same
nodes the doctor probe couldn't.

Fix: new fleetDialTarget(device) prefers the registry Tailscale dnsName/IP
(falls back to the bare name for address-less manual devices), and all three
fleet probes dial it. Also fail fast — skip the 15s+30s version+doctor for a
device the stats probe already found unreachable. Verified live: yosemite-m1..m6
went from 'probe failed / ssh: connect to host 192.168.1.101' to full healthy
doctor rows with NO change to ~/.ssh/config. 4 unit tests for fleetDialTarget.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@prix-cloud

prix-cloud Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Code Reviewer

Verdict: Ready to merge

Build:tsc typecheck clean, no errors.
Tests: ✅ 153 passed / 0 failed across 16 files, scoped to src/commands/ssh.test.ts, src/commands/check.test.ts, and src/lib/devices/** (the directories this diff touches or shares fan-out infra with) — including the 4 new fleetDialTarget tests. Scope note: I did not run the full repo-wide suite; it spends real wall-clock exercising actual agent CLI subprocesses per this repo's "real services, no mocking" convention and didn't finish in a reasonable review window. The scoped run is a faithful signal for this change's blast radius (ssh.ts, its test, and the lib/devices/ modules it calls into).

Read AGENTS.md (root), apps/cli/AGENTS.md before reviewing — build/test commands, no-mocking convention, and the fragment-based changelog format are all as documented there.

Changes that work well

  • fleetDialTarget() is a clean, pure, well-tested wrapper: it delegates to the existing sshTargetFor() (same address probeDeviceStats already dials via buildSshInvocation), so the "fail fast on stats-unreachable" optimization added in runFleetStatus is genuinely comparing the same target, not a different one — I traced this end to end (health.ts:210buildSshInvocationconnect.ts:91sshTargetFor) to confirm the PR body's claim.
  • The catch-fallback (device.user ? user@device.name : device.name) preserves prior behavior exactly for address-less manual devices — verified by the 4th test case.
  • No injection risk in the new dial path: every call site pipes target.dialTarget into sshExecAsync, which validates via assertValidSshTarget internally regardless of which branch of fleetDialTarget produced the string.
  • Changelog fragment ('.changelog/next/fleet-ssh-drift-hardening.md) follows the repo's fragment convention and accurately describes the change.
  • Test fixtures (100.74.242.106, tail1a85a1.ts.net) match the same placeholder Tailscale addresses used across ~10 other existing test files in the repo — not a secret leak.

Things to verify manually

  • runFleetStatus's new fail-fast path treats statsMap.get(name)?.reachable === false as authoritative to skip the version/doctor probes entirely. loadFleetStats is cache-first (daemon-warmed every ~3 min per stats-cache.ts's module doc) unless --refresh/--live is passed — so on a non-refresh fleet status, a box that flipped back online between the last cache warm and this invocation will now be skipped straight to an "unreachable" row without a live attempt, where previously it would have gotten a real 15s+30s try. Probably an acceptable trade (matches the existing cache-first design elsewhere in this file), but worth a conscious note since it's a new false-negative surface, not just a speed win.
  • Unrelated to this PR's stated scope (which explicitly names the three fleet probes it touches), but same bug class: src/commands/check.ts:159 (probeDeviceCheck, used by agents check --devices) still dials target.name — the bare device name — through sshExecAsync, so it remains exposed to the exact stale-~/.ssh/config-shadowing issue this PR fixes elsewhere. Worth a follow-up ticket if not already tracked.
  • gh pr checks shows gitleaks failed — I pulled the job log and it's a GitHub API 503 (No server is currently available to service your request) fetching PR commits, not an actual secrets match. Infra flake, unrelated to this diff; rerun should clear it.

Reviewed by Code Reviewer — actually ran the build and tests on this branch.

…on fresh stats

- Move fleetDialTarget into lib/devices/connect.ts (next to sshTargetFor) so it's
  shared, not command-local; tests move to connect.test.ts.
- Extend the fix to agents check --devices (check.ts probeDeviceCheck), which had
  the identical bare-name bug prix flagged — now dials the registry address too.
  Verified live: 'check --devices' reaches all 12 (drift 12 of 12) where m1-m6
  previously errored on the stale ~/.ssh/config IP.
- Gate the fail-fast skip on forceRefresh (--refresh/--live): loadFleetStats is
  cache-first, so on a default run a box that came back online since the last
  daemon warm would be skipped to 'unreachable' without a live try — a false
  negative prix flagged. Now default runs always do the (now-fast) live probe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@muqsitnawaz

Copy link
Copy Markdown
Contributor Author

Thanks — both notes addressed in 1f56675 (even though the verdict was ready-to-merge, both were worth doing):

  1. Fail-fast false-negative — gated the skip on forceRefresh (--refresh/--live). loadFleetStats is cache-first, so on a default run a box that came back online since the last daemon warm would've been skipped to 'unreachable' without a live try. Now default runs always do the (now-fast) live probe; the skip only applies when the stats are freshly probed.

  2. check.ts:159 same bug class — fixed here rather than deferred, since this PR's promise is exactly 'a stale ssh_config can't hide a live box.' agents check --devices now dials the registry address too. Moved fleetDialTarget into lib/devices/connect.ts (next to sshTargetFor) so both commands share it; tests moved to connect.test.ts. Verified live: check --devices reaches all 12 (drift 12 of 12) where m1-m6 previously errored on the stale LAN IP.

Build clean, 16 tests pass (3 new fleetDialTarget cases). The gitleaks red earlier was a GitHub 503 outage, now green.

@muqsitnawaz

Copy link
Copy Markdown
Contributor Author

Merging on green CI (8/8) + non-author review. prix-cloud verdict was 'Ready to merge' on the core; its two non-blocking notes were then addressed in 1f56675 (gate fail-fast on --refresh; extend the fix to agents check --devices). prix did not re-review the follow-up within the window, so an independent non-author reviewer verified the 1f56675 delta end-to-end (helper moved to lib/devices/connect.ts, check.ts dials the registry address with no bare-name sites left, fail-fast gated on forceRefresh, tests non-tautological, tsc clean). Rebase-merging.

@muqsitnawaz
muqsitnawaz merged commit 7a81b8d into main Jul 20, 2026
8 checks passed
@muqsitnawaz
muqsitnawaz deleted the fleet-ssh-hardening branch July 20, 2026 04:22
@prix-cloud

prix-cloud Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Code Reviewer

Verdict: Ready to merge

Build: tsc clean (bun run build), no errors.
Tests: bun test src/lib/devices/fleet.test.ts src/lib/devices/stats-cache.test.ts src/lib/devices/health.test.ts src/lib/devices/health-report.test.ts src/lib/devices/connect.test.ts src/commands/check.test.ts → 68 passed, 0 failed (includes the 4 new fleetDialTarget cases). A full-repo bun test run also surfaced 44 pre-existing failures in src/commands/__tests__/sessions.test.ts and beta.test.ts (session-sync / OpenClaw / sandbox-clock related) — unrelated to this diff, none of the touched files (connect.ts, ssh.ts, check.ts, fleet.ts) are anywhere near that code path, so these are pre-existing/environmental, not a regression from this PR.

Read apps/cli/AGENTS.md (build/test commands, source layout, "real services only" convention) and the root CLAUDE.md/AGENTS.md (repo map, PR/branch conventions) before reviewing.

Changes that work well

  • fleetDialTarget() (src/lib/devices/connect.ts:56) is a clean, pure wrapper around the existing sshTargetFor() — reuses the same dnsName ?? ip resolution and injection guard rather than duplicating it, and the catch fallback to the bare name genuinely can't regress address-less manual devices below their current behavior. Verified hostNameFor() (ssh-config.ts:23) really does prefer dnsName over ip, matching the new tests.
  • Confirmed the PR's central claim — that the stats probe already dials the registry address — by tracing loadFleetStatsprobeFleetStatsprobeDeviceStatsbuildSshInvocation() (connect.ts:111), which calls sshTargetFor(device) internally. So the fail-fast in runFleetStatus (ssh.ts:393) keys off a reachable: false result that came from probing the same address the new version/doctor probes now dial — a same-address correctness argument, not a guess.
  • The fail-fast is correctly gated to forceRefresh && !t.skip, so it can't mask a stale cache row as "unreachable" on a default (cache-served) run, and it doesn't stack on top of an already-offline/no-address/control skip.
  • All three fleet probes (--version, doctor --json, fleet ping) and the check.ts probe consistently switched from target.name to target.dialTarget — no missed call site.

Things to verify manually

  • src/commands/ssh.ts has no companion ssh.test.ts (pre-existing gap, not introduced by this PR — repo convention is source-adjacent tests but this file was already untested). The new fail-fast skip logic in runFleetStatus and the dialTarget wiring in probeRemoteHealth/probeRemoteAuth/runFleetPing are therefore only covered indirectly (via fleetDialTarget's own unit tests) rather than by a test that exercises runFleetStatus end-to-end. The PR's live-fleet verification (yosemite-m1..m6) is good evidence but isn't captured as a regression test — worth a follow-up if this area gets touched again.

Reviewed by Code Reviewer — actually ran the build and tests on this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant